| Message ID | 20201211170413.2269479-1-idosch@idosch.org |
|---|---|
| Headers | show |
| Series | mlxsw: Introduce initial XM router support | expand |
On Fri, 11 Dec 2020 19:04:11 +0200 Ido Schimmel wrote: > From: Jiri Pirko <jiri@nvidia.com> > > Upon route insertion and removal, it is needed to flush possibly cached > entries from the XM cache. Extend XM op context to carry information > needed for the flush. Implement the flush in delayed work since for HW > design reasons there is a need to wait 50usec before the flush can be > done. If during this time comes the same flush request, consolidate it > to the first one. Implement this queued flushes by a hashtable. > > Signed-off-by: Jiri Pirko <jiri@nvidia.com> > Signed-off-by: Ido Schimmel <idosch@nvidia.com> 32 bit does not like this patch: In file included from ../include/linux/bitops.h:5, from ../include/linux/kernel.h:12, from ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:4: ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c: In function ‘mlxsw_sp_router_xm_flush_mask4’: ../include/linux/bits.h:36:11: warning: right shift count is negative [-Wshift-count-negative] 36 | (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) | ^~ ../include/linux/bits.h:38:31: note: in expansion of macro ‘__GENMASK’ 38 | (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l)) | ^~~~~~~~~ ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:9: note: in expansion of macro ‘GENMASK’ 361 | return GENMASK(32, 32 - prefix_len); | ^~~~~~~ ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:16: warning: shift count is negative (-1) In file included from ../include/linux/bitops.h:5, from ../include/linux/kernel.h:12, from ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:4: ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c: In function ‘mlxsw_sp_router_xm_flush_mask4’: ../include/linux/bits.h:36:11: warning: right shift count is negative [-Wshift-count-negative] 36 | (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) | ^~ ../include/linux/bits.h:38:31: note: in expansion of macro ‘__GENMASK’ 38 | (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l)) | ^~~~~~~~~ ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:9: note: in expansion of macro ‘GENMASK’ 361 | return GENMASK(32, 32 - prefix_len); | ^~~~~~~
On Fri, Dec 11, 2020 at 08:24:27PM -0800, Jakub Kicinski wrote: > On Fri, 11 Dec 2020 19:04:11 +0200 Ido Schimmel wrote: > > From: Jiri Pirko <jiri@nvidia.com> > > > > Upon route insertion and removal, it is needed to flush possibly cached > > entries from the XM cache. Extend XM op context to carry information > > needed for the flush. Implement the flush in delayed work since for HW > > design reasons there is a need to wait 50usec before the flush can be > > done. If during this time comes the same flush request, consolidate it > > to the first one. Implement this queued flushes by a hashtable. > > > > Signed-off-by: Jiri Pirko <jiri@nvidia.com> > > Signed-off-by: Ido Schimmel <idosch@nvidia.com> > > 32 bit does not like this patch: Thanks Jiri, looks like this fix is needed: diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c index b680c22eff7d..d213af723a2a 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c @@ -358,7 +358,7 @@ mlxsw_sp_router_xm_cache_flush_node_destroy(struct mlxsw_sp *mlxsw_sp, static u32 mlxsw_sp_router_xm_flush_mask4(u8 prefix_len) { - return GENMASK(32, 32 - prefix_len); + return GENMASK(31, 32 - prefix_len); } static unsigned char *mlxsw_sp_router_xm_flush_mask6(u8 prefix_len) > > In file included from ../include/linux/bitops.h:5, > from ../include/linux/kernel.h:12, > from ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:4: > ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c: In function ‘mlxsw_sp_router_xm_flush_mask4’: > ../include/linux/bits.h:36:11: warning: right shift count is negative [-Wshift-count-negative] > 36 | (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) > | ^~ > ../include/linux/bits.h:38:31: note: in expansion of macro ‘__GENMASK’ > 38 | (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l)) > | ^~~~~~~~~ > ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:9: note: in expansion of macro ‘GENMASK’ > 361 | return GENMASK(32, 32 - prefix_len); > | ^~~~~~~ > ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:16: warning: shift count is negative (-1) > In file included from ../include/linux/bitops.h:5, > from ../include/linux/kernel.h:12, > from ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:4: > ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c: In function ‘mlxsw_sp_router_xm_flush_mask4’: > ../include/linux/bits.h:36:11: warning: right shift count is negative [-Wshift-count-negative] > 36 | (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) > | ^~ > ../include/linux/bits.h:38:31: note: in expansion of macro ‘__GENMASK’ > 38 | (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l)) > | ^~~~~~~~~ > ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:9: note: in expansion of macro ‘GENMASK’ > 361 | return GENMASK(32, 32 - prefix_len); > | ^~~~~~~
Sat, Dec 12, 2020 at 06:25:02PM CET, idosch@idosch.org wrote: >On Fri, Dec 11, 2020 at 08:24:27PM -0800, Jakub Kicinski wrote: >> On Fri, 11 Dec 2020 19:04:11 +0200 Ido Schimmel wrote: >> > From: Jiri Pirko <jiri@nvidia.com> >> > >> > Upon route insertion and removal, it is needed to flush possibly cached >> > entries from the XM cache. Extend XM op context to carry information >> > needed for the flush. Implement the flush in delayed work since for HW >> > design reasons there is a need to wait 50usec before the flush can be >> > done. If during this time comes the same flush request, consolidate it >> > to the first one. Implement this queued flushes by a hashtable. >> > >> > Signed-off-by: Jiri Pirko <jiri@nvidia.com> >> > Signed-off-by: Ido Schimmel <idosch@nvidia.com> >> >> 32 bit does not like this patch: > >Thanks > >Jiri, looks like this fix is needed: Okay, will send you fixed version. Thx. > >diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c >index b680c22eff7d..d213af723a2a 100644 >--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c >+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c >@@ -358,7 +358,7 @@ mlxsw_sp_router_xm_cache_flush_node_destroy(struct mlxsw_sp *mlxsw_sp, > > static u32 mlxsw_sp_router_xm_flush_mask4(u8 prefix_len) > { >- return GENMASK(32, 32 - prefix_len); >+ return GENMASK(31, 32 - prefix_len); > } > > static unsigned char *mlxsw_sp_router_xm_flush_mask6(u8 prefix_len) > >> >> In file included from ../include/linux/bitops.h:5, >> from ../include/linux/kernel.h:12, >> from ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:4: >> ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c: In function ‘mlxsw_sp_router_xm_flush_mask4’: >> ../include/linux/bits.h:36:11: warning: right shift count is negative [-Wshift-count-negative] >> 36 | (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) >> | ^~ >> ../include/linux/bits.h:38:31: note: in expansion of macro ‘__GENMASK’ >> 38 | (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l)) >> | ^~~~~~~~~ >> ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:9: note: in expansion of macro ‘GENMASK’ >> 361 | return GENMASK(32, 32 - prefix_len); >> | ^~~~~~~ >> ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:16: warning: shift count is negative (-1) >> In file included from ../include/linux/bitops.h:5, >> from ../include/linux/kernel.h:12, >> from ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:4: >> ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c: In function ‘mlxsw_sp_router_xm_flush_mask4’: >> ../include/linux/bits.h:36:11: warning: right shift count is negative [-Wshift-count-negative] >> 36 | (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) >> | ^~ >> ../include/linux/bits.h:38:31: note: in expansion of macro ‘__GENMASK’ >> 38 | (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l)) >> | ^~~~~~~~~ >> ../drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c:361:9: note: in expansion of macro ‘GENMASK’ >> 361 | return GENMASK(32, 32 - prefix_len); >> | ^~~~~~~
From: Ido Schimmel <idosch@nvidia.com> This patch set implements initial eXtended Mezzanine (XM) router support. The XM is an external device connected to the Spectrum-{2,3} ASICs using dedicated Ethernet ports. Its purpose is to increase the number of routes that can be offloaded to hardware. This is achieved by having the ASIC act as a cache that refers cache misses to the XM where the FIB is stored and LPM lookup is performed. Future patch sets will add more sophisticated cache flushing and selftests that utilize cache counters on the ASIC, which we plan to expose via devlink-metric [1]. Patch set overview: Patches #1-#2 add registers to insert/remove routes to/from the XM and to enable/disable it. Patch #3 utilizes these registers in order to implement XM-specific router low-level operations. Patches #4-#5 query from firmware the availability of the XM and the local ports that are used to connect the ASIC to the XM, so that netdevs will not be created for them. Patches #6-#8 initialize the XM by configuring its cache parameters. Patch #9-#10 implement cache management, so that LPM lookup will be correctly cached in the ASIC. Patches #11-#13 implement cache flushing, so that routes insertions/removals to/from the XM will flush the affected entries in the cache. Patch #14 configures the ASIC to allocate half of its memory for the cache, so that room will be left for other entries (e.g., FDBs, neighbours). Patch #15 starts using the XM for IPv4 route offload, when available. [1] https://lore.kernel.org/netdev/20200817125059.193242-1-idosch@idosch.org/ Jiri Pirko (15): mlxsw: reg: Add XM Direct Register mlxsw: reg: Add Router XLT Enable Register mlxsw: spectrum_router: Introduce XM implementation of router low-level ops mlxsw: pci: Obtain info about ports used by eXtended mezanine mlxsw: Ignore ports that are connected to eXtended mezanine mlxsw: reg: Add Router XLT M select Register mlxsw: reg: Add XM Lookup Table Query Register mlxsw: spectrum_router: Introduce per-ASIC XM initialization mlxsw: reg: Add XM Router M Table Register mlxsw: spectrum_router_xm: Implement L-value tracking for M-index mlxsw: reg: Add Router LPM Cache ML Delete Register mlxsw: reg: Add Router LPM Cache Enable Register mlxsw: spectrum_router_xm: Introduce basic XM cache flushing mlxsw: spectrum: Set KVH XLT cache mode for Spectrum2/3 mlxsw: spectrum_router: Use eXtended mezzanine to offload IPv4 router drivers/net/ethernet/mellanox/mlxsw/Makefile | 1 + drivers/net/ethernet/mellanox/mlxsw/cmd.h | 30 + drivers/net/ethernet/mellanox/mlxsw/core.c | 12 + drivers/net/ethernet/mellanox/mlxsw/core.h | 12 +- drivers/net/ethernet/mellanox/mlxsw/minimal.c | 3 +- drivers/net/ethernet/mellanox/mlxsw/pci.c | 33 +- drivers/net/ethernet/mellanox/mlxsw/reg.h | 585 ++++++++++++- .../net/ethernet/mellanox/mlxsw/spectrum.c | 5 + .../ethernet/mellanox/mlxsw/spectrum_router.c | 23 +- .../ethernet/mellanox/mlxsw/spectrum_router.h | 10 + .../mellanox/mlxsw/spectrum_router_xm.c | 812 ++++++++++++++++++ 11 files changed, 1518 insertions(+), 8 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlxsw/spectrum_router_xm.c