| Message ID | 1488993907-13037-1-git-send-email-ross.burton@intel.com |
|---|---|
| State | New |
| Headers | show |
One of the uses I've seen people ask for is to take the (patched) versions that OE puts together, store these in their own git trees and manage it there -- also allowing them to patch them directly in their own trees. It appears to me this could be used to pull down that content. Have you thought about adding anything to make it easier for someone to generate the trees? --Mark On 3/8/17 11:25 AM, Ross Burton wrote: > This class lets you use BBCLASSEXTEND to add a variant of the recipe that > fetches from git instead of a tarball. > > For example: > > BBCLASSEXTEND = "gitupstream:target" > SRC_URI_class-gitupstream = "git://git.example.com/example" > SRCREV_class-gitupstream = "abcd1234" > > This variant will have DEFAULT_PREFERENCE set to -1 so it needs to be selected > to be used, and any git-specific tweaks can be done with the class-gitupstream > override, for example: > > DEPENDS_append_class-gitupstream = " gperf-native" > > do_configure_prepend_class-gitupstream() { > touch ${S}/README > } > > It currently only supports creating a git variant of the target recipe, not > native or nativesdk. The BBCLASSEXTEND syntax (gitupstream:target) was chosen > so that support for native and nativesdk can be added at a later date. > > [ YOCTO #10215 ] > > Signed-off-by: Ross Burton <ross.burton@intel.com> > --- > meta/classes/gitupstream.bbclass | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > create mode 100644 meta/classes/gitupstream.bbclass > > diff --git a/meta/classes/gitupstream.bbclass b/meta/classes/gitupstream.bbclass > new file mode 100644 > index 0000000..405283d > --- /dev/null > +++ b/meta/classes/gitupstream.bbclass > @@ -0,0 +1,25 @@ > +CLASSOVERRIDE = "class-gitupstream" > + > +# TODO doesn't let you gitupstream a native recipe yet > + > +python gitupstream_virtclass_handler () { > + # Do nothing if this is inherited, as it's for BBCLASSEXTEND > + if "gitupstream" not in (e.data.getVar('BBCLASSEXTEND', True) or ""): > + bb.error("Don't inherit gitupstream, use BBCLASSEXTEND") > + return > + > + variant = d.getVar("BBEXTENDVARIANT", True) > + if variant not in ("target"): > + bb.error("Pass the variant when using gitupstream, for example gitupstream:target") > + > + d.setVar("DEFAULT_PREFERENCE", "-1") > + d.setVar("S", "${WORKDIR}/git") > + > + # Modify the SRCREV, if the recipe hasn't used PV_class-gitupstream to > + # assign a +gitX PV already. > + pv = d.getVar("PV", True) > + if "+git" not in pv: > + d.setVar("PV", pv + "+git${SRCPV}") > +} > +addhandler gitupstream_virtclass_handler > +gitupstream_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" > -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 8 March 2017 at 17:31, Mark Hatle <mark.hatle@windriver.com> wrote: > One of the uses I've seen people ask for is to take the (patched) versions > that > OE puts together, store these in their own git trees and manage it there > -- also > allowing them to patch them directly in their own trees. > > It appears to me this could be used to pull down that content. Have you > thought > about adding anything to make it easier for someone to generate the trees? > That sounds a lot like what devtool lets you do already. Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 3/8/17 11:36 AM, Burton, Ross wrote: > > On 8 March 2017 at 17:31, Mark Hatle <mark.hatle@windriver.com > <mailto:mark.hatle@windriver.com>> wrote: > > One of the uses I've seen people ask for is to take the (patched) versions that > OE puts together, store these in their own git trees and manage it there -- also > allowing them to patch them directly in their own trees. > > It appears to me this could be used to pull down that content. Have you thought > about adding anything to make it easier for someone to generate the trees? > > > That sounds a lot like what devtool lets you do already. Devtool can do something similar, create a custom recipe... but I don't believe it has the ability to take an existing (patched) recipe, create a git tree somewhere [external], and then make it available for modification. The request I keep getting is that they want a set of git repositories (on their own SCM system) that they can directly make changes to, updating specific sources. (They do not want to change the existing recipes, other then to specify this magic SCM system of theirs as where to pull the [already patched] source from.) I see how this patch can be used to point an existing recipe to one of those 'local SCM systems'... but creating the repository would still be an exercise for the user. --Mark > Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 8 March 2017 at 17:43, Mark Hatle <mark.hatle@windriver.com> wrote: > Devtool can do something similar, create a custom recipe... but I don't > believe > it has the ability to take an existing (patched) recipe, create a git tree > somewhere [external], and then make it available for modification. > devtool modify RECIPE will generate this git repository for you, so doing that and then copying the git repo somewhere reachable seems like it mostly solves the problem? Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 3/8/17 11:45 AM, Burton, Ross wrote: > > On 8 March 2017 at 17:43, Mark Hatle <mark.hatle@windriver.com > <mailto:mark.hatle@windriver.com>> wrote: > > Devtool can do something similar, create a custom recipe... but I don't believe > it has the ability to take an existing (patched) recipe, create a git tree > somewhere [external], and then make it available for modification. > > > devtool modify RECIPE will generate this git repository for you, so doing that > and then copying the git repo somewhere reachable seems like it mostly solves > the problem? Ahh I did not know that modify did that. Thanks! --Mark > Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Wed, Mar 8, 2017 at 9:25 AM, Ross Burton <ross.burton@intel.com> wrote: > This class lets you use BBCLASSEXTEND to add a variant of the recipe that > fetches from git instead of a tarball. Looks good. This approach could perhaps also cover git -vs- externalsrc variants of a recipe, which seems to be a big topic for a lot of users. > For example: > > BBCLASSEXTEND = "gitupstream:target" > SRC_URI_class-gitupstream = "git://git.example.com/example" > SRCREV_class-gitupstream = "abcd1234" > > This variant will have DEFAULT_PREFERENCE set to -1 so it needs to be selected > to be used So if I want to select the gitupstream variant, what should I set PREFERRED_VERSION to? >, and any git-specific tweaks can be done with the class-gitupstream > override, for example: > > DEPENDS_append_class-gitupstream = " gperf-native" > > do_configure_prepend_class-gitupstream() { > touch ${S}/README > } > > It currently only supports creating a git variant of the target recipe, not > native or nativesdk. The BBCLASSEXTEND syntax (gitupstream:target) was chosen > so that support for native and nativesdk can be added at a later date. > > [ YOCTO #10215 ] > > Signed-off-by: Ross Burton <ross.burton@intel.com> > --- > meta/classes/gitupstream.bbclass | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > create mode 100644 meta/classes/gitupstream.bbclass > > diff --git a/meta/classes/gitupstream.bbclass b/meta/classes/gitupstream.bbclass > new file mode 100644 > index 0000000..405283d > --- /dev/null > +++ b/meta/classes/gitupstream.bbclass > @@ -0,0 +1,25 @@ > +CLASSOVERRIDE = "class-gitupstream" > + > +# TODO doesn't let you gitupstream a native recipe yet > + > +python gitupstream_virtclass_handler () { > + # Do nothing if this is inherited, as it's for BBCLASSEXTEND > + if "gitupstream" not in (e.data.getVar('BBCLASSEXTEND', True) or ""): What's the significance of e.data.getVar() here -vs- d.getVar() below? (I ask because I've tried writing a few RecipePreFinalise event handlers and they don't always work quite as expected, so maybe I've been getting this wrong...). > + bb.error("Don't inherit gitupstream, use BBCLASSEXTEND") > + return > + > + variant = d.getVar("BBEXTENDVARIANT", True) Maybe drop the ", True" ? > + if variant not in ("target"): > + bb.error("Pass the variant when using gitupstream, for example gitupstream:target") > + > + d.setVar("DEFAULT_PREFERENCE", "-1") > + d.setVar("S", "${WORKDIR}/git") > + > + # Modify the SRCREV, if the recipe hasn't used PV_class-gitupstream to > + # assign a +gitX PV already. > + pv = d.getVar("PV", True) > + if "+git" not in pv: > + d.setVar("PV", pv + "+git${SRCPV}") > +} > +addhandler gitupstream_virtclass_handler > +gitupstream_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" > -- > 2.8.1 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Wed, Mar 8, 2017 at 12:54 PM, Andre McCurdy <armccurdy@gmail.com> wrote: > On Wed, Mar 8, 2017 at 9:25 AM, Ross Burton <ross.burton@intel.com> wrote: >> This class lets you use BBCLASSEXTEND to add a variant of the recipe that >> fetches from git instead of a tarball. > > Looks good. This approach could perhaps also cover git -vs- > externalsrc variants of a recipe, which seems to be a big topic for a > lot of users. > >> For example: >> >> BBCLASSEXTEND = "gitupstream:target" >> SRC_URI_class-gitupstream = "git://git.example.com/example" >> SRCREV_class-gitupstream = "abcd1234" >> >> This variant will have DEFAULT_PREFERENCE set to -1 so it needs to be selected >> to be used > > So if I want to select the gitupstream variant, what should I set > PREFERRED_VERSION to? > >>, and any git-specific tweaks can be done with the class-gitupstream >> override, for example: >> >> DEPENDS_append_class-gitupstream = " gperf-native" >> >> do_configure_prepend_class-gitupstream() { >> touch ${S}/README >> } >> >> It currently only supports creating a git variant of the target recipe, not >> native or nativesdk. The BBCLASSEXTEND syntax (gitupstream:target) was chosen >> so that support for native and nativesdk can be added at a later date. >> >> [ YOCTO #10215 ] >> >> Signed-off-by: Ross Burton <ross.burton@intel.com> >> --- >> meta/classes/gitupstream.bbclass | 25 +++++++++++++++++++++++++ >> 1 file changed, 25 insertions(+) >> create mode 100644 meta/classes/gitupstream.bbclass >> >> diff --git a/meta/classes/gitupstream.bbclass b/meta/classes/gitupstream.bbclass >> new file mode 100644 >> index 0000000..405283d >> --- /dev/null >> +++ b/meta/classes/gitupstream.bbclass >> @@ -0,0 +1,25 @@ >> +CLASSOVERRIDE = "class-gitupstream" >> + >> +# TODO doesn't let you gitupstream a native recipe yet >> + >> +python gitupstream_virtclass_handler () { >> + # Do nothing if this is inherited, as it's for BBCLASSEXTEND >> + if "gitupstream" not in (e.data.getVar('BBCLASSEXTEND', True) or ""): > > What's the significance of e.data.getVar() here -vs- d.getVar() below? To answer my own question, this seems to be covered in the bitbake user manual: "The global datastore is available as "d". In legacy code, you might see "e.data" used to get the datastore. However, realize that "e.data" is deprecated and you should use "d" going forward." http://www.yoctoproject.org/docs/2.2.1/bitbake-user-manual/bitbake-user-manual.html#events > (I ask because I've tried writing a few RecipePreFinalise event > handlers and they don't always work quite as expected, so maybe I've > been getting this wrong...). > >> + bb.error("Don't inherit gitupstream, use BBCLASSEXTEND") >> + return >> + >> + variant = d.getVar("BBEXTENDVARIANT", True) > > Maybe drop the ", True" ? > >> + if variant not in ("target"): >> + bb.error("Pass the variant when using gitupstream, for example gitupstream:target") >> + >> + d.setVar("DEFAULT_PREFERENCE", "-1") >> + d.setVar("S", "${WORKDIR}/git") >> + >> + # Modify the SRCREV, if the recipe hasn't used PV_class-gitupstream to >> + # assign a +gitX PV already. >> + pv = d.getVar("PV", True) >> + if "+git" not in pv: >> + d.setVar("PV", pv + "+git${SRCPV}") >> +} >> +addhandler gitupstream_virtclass_handler >> +gitupstream_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" >> -- >> 2.8.1 >> >> -- >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Wed, 2017-03-08 at 14:44 -0800, Andre McCurdy wrote: > On Wed, Mar 8, 2017 at 12:54 PM, Andre McCurdy <armccurdy@gmail.com> > wrote: > > > > On Wed, Mar 8, 2017 at 9:25 AM, Ross Burton <ross.burton@intel.com> > > wrote: > > > > > > This class lets you use BBCLASSEXTEND to add a variant of the > > > recipe that > > > fetches from git instead of a tarball. > > Looks good. This approach could perhaps also cover git -vs- > > externalsrc variants of a recipe, which seems to be a big topic for > > a > > lot of users. > > > > > > > > For example: > > > > > > BBCLASSEXTEND = "gitupstream:target" > > > SRC_URI_class-gitupstream = "git://git.example.com/example" > > > SRCREV_class-gitupstream = "abcd1234" > > > > > > This variant will have DEFAULT_PREFERENCE set to -1 so it needs > > > to be selected > > > to be used > > So if I want to select the gitupstream variant, what should I set > > PREFERRED_VERSION to? > > > > > > > > , and any git-specific tweaks can be done with the class- > > > gitupstream > > > override, for example: > > > > > > DEPENDS_append_class-gitupstream = " gperf-native" > > > > > > do_configure_prepend_class-gitupstream() { > > > touch ${S}/README > > > } > > > > > > It currently only supports creating a git variant of the target > > > recipe, not > > > native or nativesdk. The BBCLASSEXTEND syntax > > > (gitupstream:target) was chosen > > > so that support for native and nativesdk can be added at a later > > > date. > > > > > > [ YOCTO #10215 ] > > > > > > Signed-off-by: Ross Burton <ross.burton@intel.com> > > > --- > > > meta/classes/gitupstream.bbclass | 25 +++++++++++++++++++++++++ > > > 1 file changed, 25 insertions(+) > > > create mode 100644 meta/classes/gitupstream.bbclass > > > > > > diff --git a/meta/classes/gitupstream.bbclass > > > b/meta/classes/gitupstream.bbclass > > > new file mode 100644 > > > index 0000000..405283d > > > --- /dev/null > > > +++ b/meta/classes/gitupstream.bbclass > > > @@ -0,0 +1,25 @@ > > > +CLASSOVERRIDE = "class-gitupstream" > > > + > > > +# TODO doesn't let you gitupstream a native recipe yet > > > + > > > +python gitupstream_virtclass_handler () { > > > + # Do nothing if this is inherited, as it's for BBCLASSEXTEND > > > + if "gitupstream" not in (e.data.getVar('BBCLASSEXTEND', > > > True) or ""): > > What's the significance of e.data.getVar() here -vs- d.getVar() > > below? > To answer my own question, this seems to be covered in the bitbake > user manual: > > "The global datastore is available as "d". In legacy code, you might > see "e.data" used to get the datastore. However, realize that > "e.data" > is deprecated and you should use "d" going forward." > > http://www.yoctoproject.org/docs/2.2.1/bitbake-user-manual/bitbake-us > er-manual.html#events Well found, I was about to reply mentioning this. I did help Scott update that piece of the manual recently! :) Cheers, Richard
On 8 March 2017 at 20:54, Andre McCurdy <armccurdy@gmail.com> wrote: > So if I want to select the gitupstream variant, what should I set > PREFERRED_VERSION to? > My testing recipe is fontconfig, so I have this in my local.conf: PREFERRED_VERSION_fontconfig = "2.12.1+git%" Yes, you need to keep the versions in track with the version bumps. I'll sent a v2 with the other fixes in shortly. Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 8 March 2017 at 17:25, Ross Burton <ross.burton@intel.com> wrote: > + d.setVar("S", "${WORKDIR}/git") > + d.setVar("PV", pv + "+git${SRCPV}") > I guess these are the only two lines which are git-specific. Would it be sensible to be VCS-agnostic and simply look at the protocol of the first entry in SRC_URI? So if someone had: BBCLASSEXTEND="devupstream:target" SRC_URI_class-devupstream = "svn://some/svn/url" You'd get: S=${WORKDIR}/svn PV=${PV}+svn${SRCPV} Annoyingly that breaks current behaviour as svn checkouts use the module name, whereas git clones just use "git" (and I can't remember why that is). The class could look at the SRC_URI further: for git set SRC_URI to WORKDIR/git, for svn set WORKDIR/{module name}. Also current practise is +svnr{SRCPV} not +svn, so that could either be special-cased again or ignored. Thoughts? I don't want something to be merged which we rewrite in a months' time... Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Thu, 2017-03-09 at 12:38 +0000, Burton, Ross wrote: > > On 8 March 2017 at 17:25, Ross Burton <ross.burton@intel.com> wrote: > > + d.setVar("S", "${WORKDIR}/git") > > + d.setVar("PV", pv + "+git${SRCPV}") > > > I guess these are the only two lines which are git-specific. Would > it be sensible to be VCS-agnostic and simply look at the protocol of > the first entry in SRC_URI? > > So if someone had: > > BBCLASSEXTEND="devupstream:target" > SRC_URI_class-devupstream = "svn://some/svn/url" > > You'd get: > > S=${WORKDIR}/svn > PV=${PV}+svn${SRCPV} > > Annoyingly that breaks current behaviour as svn checkouts use the > module name, whereas git clones just use "git" (and I can't remember > why that is). The class could look at the SRC_URI further: for git > set SRC_URI to WORKDIR/git, for svn set WORKDIR/{module name}. Also > current practise is +svnr{SRCPV} not +svn, so that could either be > special-cased again or ignored. > > Thoughts? I don't want something to be merged which we rewrite in a > months' time... I worry that making it support generic scms may make it fragile and hard to extend if we do end up needing to do scm specific work in here so I'm torn. Having a class per scm would also be annoying... Cheers, Richard
On 9 March 2017 at 13:03, Richard Purdie <richard.purdie@linuxfoundation.org > wrote: > I worry that making it support generic scms may make it fragile and > hard to extend if we do end up needing to do scm specific work in here > so I'm torn. Having a class per scm would also be annoying... > So I have this so far: CLASSOVERRIDE = "class-gitupstream" # TODO doesn't let you gitupstream a native recipe yet python gitupstream_virtclass_handler () { # Do nothing if this is inherited, as it's for BBCLASSEXTEND if "gitupstream" not in (d.getVar('BBCLASSEXTEND') or ""): bb.error("Don't inherit gitupstream, use BBCLASSEXTEND") return variant = d.getVar("BBEXTENDVARIANT") if variant not in ("target"): bb.error("Pass the variant when using gitupstream, for example gitupstream:target") src_uri = d.getVar("SRC_URI").split()[0] uri = bb.fetch2.URI(src_uri) d.setVar("DEFAULT_PREFERENCE", "-1") if uri.scheme == "git": d.setVar("S", "${WORKDIR}/git") elif uri.scheme == "svn": d.setVar("S", "${WORKDIR}/" + uri.params["module"]) # Modify the PV if the recipe hasn't already overridden it pv = d.getVar("PV") proto_marker = "+" + uri.scheme if proto_marker not in pv: d.setVar("PV", pv + proto_marker + "${SRCPV}") } addhandler gitupstream_virtclass_handler gitupstream_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" (obviously s/gitupstream/devupstream/ still needs to happen) It's useful in that it adjust S automatically for svn fetches, but the problem is that bitbake looks for svn: URIs too early and so this doesn't cause subversion-native to be built. I'm thinking I'll finish renaming this to devupstream, remove the svn specific part, and make a note about attempting to fix the fetcher problem. Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
diff --git a/meta/classes/gitupstream.bbclass b/meta/classes/gitupstream.bbclass new file mode 100644 index 0000000..405283d --- /dev/null +++ b/meta/classes/gitupstream.bbclass @@ -0,0 +1,25 @@ +CLASSOVERRIDE = "class-gitupstream" + +# TODO doesn't let you gitupstream a native recipe yet + +python gitupstream_virtclass_handler () { + # Do nothing if this is inherited, as it's for BBCLASSEXTEND + if "gitupstream" not in (e.data.getVar('BBCLASSEXTEND', True) or ""): + bb.error("Don't inherit gitupstream, use BBCLASSEXTEND") + return + + variant = d.getVar("BBEXTENDVARIANT", True) + if variant not in ("target"): + bb.error("Pass the variant when using gitupstream, for example gitupstream:target") + + d.setVar("DEFAULT_PREFERENCE", "-1") + d.setVar("S", "${WORKDIR}/git") + + # Modify the SRCREV, if the recipe hasn't used PV_class-gitupstream to + # assign a +gitX PV already. + pv = d.getVar("PV", True) + if "+git" not in pv: + d.setVar("PV", pv + "+git${SRCPV}") +} +addhandler gitupstream_virtclass_handler +gitupstream_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
This class lets you use BBCLASSEXTEND to add a variant of the recipe that fetches from git instead of a tarball. For example: BBCLASSEXTEND = "gitupstream:target" SRC_URI_class-gitupstream = "git://git.example.com/example" SRCREV_class-gitupstream = "abcd1234" This variant will have DEFAULT_PREFERENCE set to -1 so it needs to be selected to be used, and any git-specific tweaks can be done with the class-gitupstream override, for example: DEPENDS_append_class-gitupstream = " gperf-native" do_configure_prepend_class-gitupstream() { touch ${S}/README } It currently only supports creating a git variant of the target recipe, not native or nativesdk. The BBCLASSEXTEND syntax (gitupstream:target) was chosen so that support for native and nativesdk can be added at a later date. [ YOCTO #10215 ] Signed-off-by: Ross Burton <ross.burton@intel.com> --- meta/classes/gitupstream.bbclass | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 meta/classes/gitupstream.bbclass -- 2.8.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core