[bpf-next,v2,1/6] tools: Help cross-building with clang

Cross-compilation with clang uses the -target parameter rather than a
toolchain prefix. Just like the kernel Makefile, add that parameter to
CFLAGS when CROSS_COMPILE is set.

Unlike the kernel Makefile, we use the --sysroot and --gcc-toolchain
options because unlike the kernel, tools require standard libraries.
Commit c91d4e47e10e ("Makefile: Remove '--gcc-toolchain' flag") provides
some background about --gcc-toolchain. Normally clang finds on its own
the additional utilities and libraries that it needs (for example GNU ld
or glibc). On some systems however, this autodetection doesn't work.
There, our only recourse is asking GCC directly, and pass the result to
--sysroot and --gcc-toolchain. Of course that only works when a cross
GCC is available.

Autodetection worked fine on Debian, but to use the aarch64-linux-gnu
toolchain from Archlinux I needed both --sysroot (for crt1.o) and
--gcc-toolchain (for crtbegin.o, -lgcc). The --prefix parameter wasn't
needed there, but it might be useful on other distributions.

Use the CLANG_CROSS_FLAGS variable instead of CLANG_FLAGS because it
allows tools such as bpftool, that need to build both host and target
binaries, to easily filter out the cross-build flags from CFLAGS.

Acked-by: Quentin Monnet <quentin@isovalent.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
Most tools I looked at needed additional changes to support cross-build
with clang. I've only done the work for bpf tools.
---
 tools/scripts/Makefile.include | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

Hi Jean-Philippe,

On Thu, Dec 16, 2021 at 04:38:38PM +0000, Jean-Philippe Brucker wrote:
> Cross-compilation with clang uses the -target parameter rather than a
> toolchain prefix. Just like the kernel Makefile, add that parameter to
> CFLAGS when CROSS_COMPILE is set.
> 
> Unlike the kernel Makefile, we use the --sysroot and --gcc-toolchain
> options because unlike the kernel, tools require standard libraries.
> Commit c91d4e47e10e ("Makefile: Remove '--gcc-toolchain' flag") provides
> some background about --gcc-toolchain. Normally clang finds on its own
> the additional utilities and libraries that it needs (for example GNU ld
> or glibc). On some systems however, this autodetection doesn't work.
> There, our only recourse is asking GCC directly, and pass the result to
> --sysroot and --gcc-toolchain. Of course that only works when a cross
> GCC is available.
> 
> Autodetection worked fine on Debian, but to use the aarch64-linux-gnu
> toolchain from Archlinux I needed both --sysroot (for crt1.o) and
> --gcc-toolchain (for crtbegin.o, -lgcc). The --prefix parameter wasn't
> needed there, but it might be useful on other distributions.
> 
> Use the CLANG_CROSS_FLAGS variable instead of CLANG_FLAGS because it
> allows tools such as bpftool, that need to build both host and target
> binaries, to easily filter out the cross-build flags from CFLAGS.
> 
> Acked-by: Quentin Monnet <quentin@isovalent.com>
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> ---
> Most tools I looked at needed additional changes to support cross-build
> with clang. I've only done the work for bpf tools.
> ---
>  tools/scripts/Makefile.include | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
> index 071312f5eb92..b0be5f40a3f1 100644
> --- a/tools/scripts/Makefile.include
> +++ b/tools/scripts/Makefile.include
> @@ -87,7 +87,18 @@ LLVM_STRIP	?= llvm-strip
>  
>  ifeq ($(CC_NO_CLANG), 1)
>  EXTRA_WARNINGS += -Wstrict-aliasing=3
> -endif
> +
> +else ifneq ($(CROSS_COMPILE),)
> +CLANG_CROSS_FLAGS := --target=$(notdir $(CROSS_COMPILE:%-=%))
> +GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)gcc))

Apologies for noticing this so late, I only ran into this recently.

This line causes a warning when running 'make clean' when
'$(CROSS_COMPILE)gcc' does not exist in PATH. For example:

$ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- LLVM=1 LLVM_IAS=0 clean
which: no powerpc-linux-gnu-gcc in ($PATH)

I only have powerpc-linux-gnu binutils in my PATH, not GCC, as I am only
working with clang.

This happens because of the 'resolve_btfids_clean target', which always
runs when running the 'clean' target on an in-tree build (since
$(objtree) = $(srctree)).

I tried looking into the best way to fix this but I am not at all
familiar with the tools/ build system; would you mind taking a look?
I see some machinery at the top of tools/bpf/Makefile for avoiding
running some commands under certain commands but I am unsure how to
shuffle that around to make everything work.

Cheers,
Nathan

> +ifneq ($(GCC_TOOLCHAIN_DIR),)
> +CLANG_CROSS_FLAGS += --prefix=$(GCC_TOOLCHAIN_DIR)$(notdir $(CROSS_COMPILE))
> +CLANG_CROSS_FLAGS += --sysroot=$(shell $(CROSS_COMPILE)gcc -print-sysroot)
> +CLANG_CROSS_FLAGS += --gcc-toolchain=$(realpath $(GCC_TOOLCHAIN_DIR)/..)
> +endif # GCC_TOOLCHAIN_DIR
> +CFLAGS += $(CLANG_CROSS_FLAGS)
> +AFLAGS += $(CLANG_CROSS_FLAGS)
> +endif # CROSS_COMPILE
>  
>  # Hack to avoid type-punned warnings on old systems such as RHEL5:
>  # We should be changing CFLAGS and checking gcc version, but this
> -- 
> 2.34.1
> 
>

Hi Nathan,

On Fri, Jan 28, 2022 at 06:10:24PM -0700, Nathan Chancellor wrote:
> > diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
> > index 071312f5eb92..b0be5f40a3f1 100644
> > --- a/tools/scripts/Makefile.include
> > +++ b/tools/scripts/Makefile.include
> > @@ -87,7 +87,18 @@ LLVM_STRIP	?= llvm-strip
> >  
> >  ifeq ($(CC_NO_CLANG), 1)
> >  EXTRA_WARNINGS += -Wstrict-aliasing=3
> > -endif
> > +
> > +else ifneq ($(CROSS_COMPILE),)
> > +CLANG_CROSS_FLAGS := --target=$(notdir $(CROSS_COMPILE:%-=%))
> > +GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)gcc))
> 
> Apologies for noticing this so late, I only ran into this recently.
> 
> This line causes a warning when running 'make clean' when
> '$(CROSS_COMPILE)gcc' does not exist in PATH. For example:
> 
> $ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- LLVM=1 LLVM_IAS=0 clean
> which: no powerpc-linux-gnu-gcc in ($PATH)
> 
> I only have powerpc-linux-gnu binutils in my PATH, not GCC, as I am only
> working with clang.
> 
> This happens because of the 'resolve_btfids_clean target', which always
> runs when running the 'clean' target on an in-tree build (since
> $(objtree) = $(srctree)).
> 
> I tried looking into the best way to fix this but I am not at all
> familiar with the tools/ build system; would you mind taking a look?
> I see some machinery at the top of tools/bpf/Makefile for avoiding
> running some commands under certain commands but I am unsure how to
> shuffle that around to make everything work.

I think it's simpler than that, we should just suppress the errors from
'which'. It's fine that $(CROSS_COMPILE)gcc doesn't exist and
$(GCC_TOOLCHAIN_DIR) is empty, but 'which' should keep quiet about it.
I did test this patch with cross-build and no gcc, but Debian's 'which' is
quiet by default so I missed the error. I'll send a fix shortly.

Thanks,
Jean

On Mon, Jan 31, 2022 at 02:10:30PM +0000, Jean-Philippe Brucker wrote:
> Hi Nathan,
> 
> On Fri, Jan 28, 2022 at 06:10:24PM -0700, Nathan Chancellor wrote:
> > > diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
> > > index 071312f5eb92..b0be5f40a3f1 100644
> > > --- a/tools/scripts/Makefile.include
> > > +++ b/tools/scripts/Makefile.include
> > > @@ -87,7 +87,18 @@ LLVM_STRIP	?= llvm-strip
> > >  
> > >  ifeq ($(CC_NO_CLANG), 1)
> > >  EXTRA_WARNINGS += -Wstrict-aliasing=3
> > > -endif
> > > +
> > > +else ifneq ($(CROSS_COMPILE),)
> > > +CLANG_CROSS_FLAGS := --target=$(notdir $(CROSS_COMPILE:%-=%))
> > > +GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE)gcc))
> > 
> > Apologies for noticing this so late, I only ran into this recently.
> > 
> > This line causes a warning when running 'make clean' when
> > '$(CROSS_COMPILE)gcc' does not exist in PATH. For example:
> > 
> > $ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- LLVM=1 LLVM_IAS=0 clean
> > which: no powerpc-linux-gnu-gcc in ($PATH)
> > 
> > I only have powerpc-linux-gnu binutils in my PATH, not GCC, as I am only
> > working with clang.
> > 
> > This happens because of the 'resolve_btfids_clean target', which always
> > runs when running the 'clean' target on an in-tree build (since
> > $(objtree) = $(srctree)).
> > 
> > I tried looking into the best way to fix this but I am not at all
> > familiar with the tools/ build system; would you mind taking a look?
> > I see some machinery at the top of tools/bpf/Makefile for avoiding
> > running some commands under certain commands but I am unsure how to
> > shuffle that around to make everything work.
> 
> I think it's simpler than that, we should just suppress the errors from
> 'which'. It's fine that $(CROSS_COMPILE)gcc doesn't exist and
> $(GCC_TOOLCHAIN_DIR) is empty, but 'which' should keep quiet about it.

Ha, that's what I get for looking into something at the very end of a
long work week because that is a great and simple solution I probably
would not have seen :)

> I did test this patch with cross-build and no gcc, but Debian's 'which' is
> quiet by default so I missed the error. I'll send a fix shortly.

Ah, that is an interesting observation. TIL that Debian rolls their own
which [1], versus most other distros, which use GNU which [2].

I will keep an eye out for your patch so I can review it.

[1]: https://salsa.debian.org/debian/debianutils/-/blob/de14223e5bffe15e374a441302c528ffc1cbed57/which
[2]: https://savannah.gnu.org/projects/which/

Cheers,
Nathan