| Message ID | 20210816160717.31285-2-arkadiusz.kubalewski@intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC,net-next,1/7] ptp: Add interface for acquiring DPLL state | expand |
> > Previously there was no common interface for monitoring > > synchronization state of Digital Phase Locked Loop. > > > > Add interface through PTP ioctl subsystem for tools, as well as sysfs > > human-friendly part of the interface. > > > > enum dpll_state moved to uapi definition, it is required to have > > common definition of DPLL states in uapi. > > This has nothing to do with PTP, right? > > So why isn't it a RTNL feature instead? > > Thanks, > Richard The logic behind adding the DPLL state to the PTP subsystem is that SyncE DPLL on Network adapters, in most cases, drive the PTP timer. Having access to it in the PTP subsystem is beneficial, as Telco standards, like G.8275.1/2, require a different behavior depending on the SyncE availability and state. Multiport adapters use a single PLL to drive all ports. If we add the PLL interface to the PTP device - a tool that would implement support for Telco standards would have a single source of information about the presence and state of physical sync. Moreover, it'll open the path for implementing PLL state-aware ext_ts that would generate events only when the PLL device is locked to the incoming signals improving the quality of external TS events. I.e. an external PLL can be used to monitor incoming 1PPS signal and disable event generation when its frequency drifts off 1Hz by preset margins. If we bind it to the Network port, a tool would need to find the port that owns the PLL and read the state from a different place, which sounds suboptimal. Additionally we'll lose ability to rely on external HW to monitor external TS events. Regards Maciek
On Tue, Aug 17, 2021 at 09:41:49AM +0000, Machnikowski, Maciej wrote: > The logic behind adding the DPLL state to the PTP subsystem is that SyncE DPLL on Network adapters, in most cases, drive the PTP timer. So what? The logic in the HW has nothing to do with the proper user space interfaces. For example, we have SO_TIMESTAMPING and PHC as separate APIs, even though HW devices often implement both. > Having access to it in the PTP subsystem is beneficial, as Telco > standards, like G.8275.1/2, require a different behavior depending > on the SyncE availability and state. Right, but this does say anything about the API. > Multiport adapters use a single PLL to drive all ports. If we add > the PLL interface to the PTP device - a tool that would implement > support for Telco standards would have a single source of > information about the presence and state of physical sync. Not really. Nothing guarantees a sane mapping from MAC to PHC. In many systems, there a many of each. > Moreover, it'll open the path for implementing PLL state-aware > ext_ts that would generate events only when the PLL device is locked > to the incoming signals improving the quality of external TS > events. I.e. an external PLL can be used to monitor incoming 1PPS > signal and disable event generation when its frequency drifts off > 1Hz by preset margins. I don't see how this prevents using RTNL (or something else) as the API for the SyncE configuration. > If we bind it to the Network port, a tool would need to find the > port that owns the PLL and read the state from a different place, > which sounds suboptimal. This is exactly how it works in ptpl4 today. Some information comes from the PHC, some from RTNL (link state), some from ethtool (phc-interface mapping and time stamping abilities). > Additionally we'll lose ability to rely on external HW to monitor > external TS events. Sorry, I can't see that at all. Please do NOT tack this stuff onto the PHC subsystem just because you can. Thanks, Richard
> -----Original Message----- > From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of > Richard Cochran > Sent: Wednesday, August 18, 2021 10:03 AM > To: Machnikowski, Maciej <maciej.machnikowski@intel.com> > Cc: cong.wang@bytedance.com; arnd@arndb.de; gustavoars@kernel.org; > netdev@vger.kernel.org; linux-kernel@vger.kernel.org; > colin.king@canonical.com; intel-wired-lan@lists.osuosl.org; nikolay@nvidia.com; > linux-kselftest@vger.kernel.org; kuba@kernel.org; shuah@kernel.org; > davem@davemloft.net > Subject: Re: [Intel-wired-lan] [RFC net-next 1/7] ptp: Add interface for acquiring > DPLL state > > On Tue, Aug 17, 2021 at 09:41:49AM +0000, Machnikowski, Maciej wrote: > > > The logic behind adding the DPLL state to the PTP subsystem is that SyncE DPLL > on Network adapters, in most cases, drive the PTP timer. > > So what? The logic in the HW has nothing to do with the proper user > space interfaces. For example, we have SO_TIMESTAMPING and PHC as > separate APIs, even though HW devices often implement both. > > > Having access to it in the PTP subsystem is beneficial, as Telco > > standards, like G.8275.1/2, require a different behavior depending > > on the SyncE availability and state. > > Right, but this does say anything about the API. > > > Multiport adapters use a single PLL to drive all ports. If we add > > the PLL interface to the PTP device - a tool that would implement > > support for Telco standards would have a single source of > > information about the presence and state of physical sync. > > Not really. Nothing guarantees a sane mapping from MAC to PHC. In > many systems, there a many of each. > Well, I think the point of placing it in the PTP subsystem is that there is a sane mapping between PHC <-> DPLL. There's only one DPLL for the PHC. Thanks, Jake
> -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: Wednesday, August 18, 2021 7:03 PM > To: Machnikowski, Maciej <maciej.machnikowski@intel.com> > Cc: Kubalewski, Arkadiusz <arkadiusz.kubalewski@intel.com>; linux- > kernel@vger.kernel.org; intel-wired-lan@lists.osuosl.org; > netdev@vger.kernel.org; linux-kselftest@vger.kernel.org; Brandeburg, > Jesse <jesse.brandeburg@intel.com>; Nguyen, Anthony L > <anthony.l.nguyen@intel.com>; davem@davemloft.net; kuba@kernel.org; > shuah@kernel.org; arnd@arndb.de; nikolay@nvidia.com; > cong.wang@bytedance.com; colin.king@canonical.com; > gustavoars@kernel.org > Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state > > > Additionally we'll lose ability to rely on external HW to monitor > > external TS events. > > Sorry, I can't see that at all. > > Please do NOT tack this stuff onto the PHC subsystem just because you can. > > Thanks, > Richard OK, Let's take a step back and forget about SyncE. A PTP clock is a device that has a phase and a frequency, and its frequency can be adjusted using API calls. On the other hand, there's the physical side of the PTP clock. The PTP clock can run on cheap quartz, on the CSAC, or the PLL. The first two of them will get a clock signal from a passive device, but in the PLL case, it'll get it from an active one. If it runs on an active PLL device, you get another place that adjusts the frequency of your PTP clock. No matter what source you use as a reference for it - CSAC, SyncE, or GNSS receiver. Adding the PLL state to the PTP subsystem is just another indicator of the state of the PTP clock. The upper layer can use it, or ignored it, but it fits into the timer subsystem, as the time generated by the PTP on top will be derived from the frequency generated by the PLL. And it is applicable to both a PHC and a completely separate implementation of timer, like the one that's present in the Time Card . Regards Maciek
> -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: Thursday, August 19, 2021 5:34 PM > To: Machnikowski, Maciej <maciej.machnikowski@intel.com> > Cc: Kubalewski, Arkadiusz <arkadiusz.kubalewski@intel.com>; linux- > kernel@vger.kernel.org; intel-wired-lan@lists.osuosl.org; > netdev@vger.kernel.org; linux-kselftest@vger.kernel.org; Brandeburg, > Jesse <jesse.brandeburg@intel.com>; Nguyen, Anthony L > <anthony.l.nguyen@intel.com>; davem@davemloft.net; kuba@kernel.org; > shuah@kernel.org; arnd@arndb.de; nikolay@nvidia.com; > cong.wang@bytedance.com; colin.king@canonical.com; > gustavoars@kernel.org; Bross, Kevin <kevin.bross@intel.com>; Stanton, > Kevin B <kevin.b.stanton@intel.com>; Ahmad Byagowi <abyagowi@fb.com> > Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state > > On Wed, Aug 18, 2021 at 10:36:03PM +0000, Machnikowski, Maciej wrote: > > > OK, Let's take a step back and forget about SyncE. > > Ahem, the title of this series is: > > [RFC net-next 0/7] Add basic SyncE interfaces > > I'd be happy to see support for configuring SyncE. > > But I guess this series is about something totally different. > > > Thanks, > Richard If it helps we'd be happy to separate that in 2 separate RFCs. This was squashed together under SyncE support umbrella to show one of the use cases, but PTP changes are more generic and cover all PTP clocks that use DPLL for the physical clock generation. Regards Maciek
On Thu, Aug 19, 2021 at 03:40:22PM +0000, Machnikowski, Maciej wrote:
> If it helps we'd be happy to separate that in 2 separate RFCs.
It would help me if you could explain the connection. I have a
totally different understanding of SyncE which I explained here:
https://lore.kernel.org/netdev/20150317161128.GA8793@localhost.localdomain/
So you need to implement two things, one in kernel and one in user
space.
1. Control bits according to IEEE 802.3 Section 40.5.2 as Ethtool or RTNL.
2. User space Ethernet Synchronization Messaging Channel (ESMC)
service according to IEEE 802.3ay
The PHY should be automatically controlled by #1.
As I said before, none of this belongs in the PHC subsystem.
Thanks,
Richard
On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote: > I did a talk at netDev 0x15 covering SyncE - you can refer to the slides for more detailed info, and hopefully the recording will be available soon as well: > https://netdevconf.info/0x15/session.html?Introduction-to-time-synchronization-over-Ethernet These slides are very clear and nicely done! ( And they also confirm that (ab)using the PHC chardev ioctl for the DPLL stuff is the wrong interface ;^) > The SyncE capable PHY is a PHY that can recover the physical clock, > at which the data symbols are transferred, (usually) divide it and > output it to the external PLL. It can also redirect the recovered > and divided clock to more than one pin. Right, and as your slides show so clearly, the DPLL is connected to the PHY, not to the time stamping unit with the PTP clock. > Since the 40.5.2 is not applicable to higher-speed ethernet which > don't use auto-negotiation, but rather the link training sequence > where the RX side always syncs its clock to the TX side. I really want an interface that will also work with Gigabit and even 100 Megabit like the PHYTER (which does support SyncE). > The external DPLL tunes the frequency generated by a crystal to the frequency recovered by the PHY, and drives the outputs. > > On the other end - the SyncE PHY uses the clock generated by the DPLL to transmit the data to the next element. So I guess that this is an implementation detail of the higher speed PHYs. > That's why the RFC proposes 2 interfaces: > - one for enabling redirected clock on a selected pin of the PHY > - one for the physical frequency lock of the DPLL > > The connection with the PTP subsystem is that in most use cases I > heard about SyncE is used as a physical frequency syntonization for > PTP clocks. As your slides correctly show, SyncE is about distributing frequency and not about Phase/ToD. Of course you can combine SyncE with PTP to get both, provided that you disable frequency adjustment in the PTP software stack (in linuxptp, this is the "nullf" servo). But SyncE in fact predates PTP, and it can and should be configurable even on interfaces that lack PHC altogther (or on kernels without PHC enabled). > Let me know if that makes more sense now. We could add a separate > SyncE and separate PTP DPLL monitoring interfaces, but in most cases > they will point to the same device. This is just a coincidence of the device you are working with. The kernel really needs an interface that works with all kind of hardware setups. Imagine a computer with discrete MACs with HW time stamping/PHC and discrete PHYs with SyncE support. The PHC driver won't have any connection to the PHY+DPLL. Your API must be on the interface/MAC, with the possibility being handled directly by the MAC driver (for integrated devices) or calling into the PHY layers (phylib, phylink, and drivers/phy). If you need a DPLL monitoring interface for your card, it ought to go through the network interface to the MAC/PHY driver and then to the DPLL itself. That way, it will work with different types of hardware. Thanks, Richard
On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote: > Since the 40.5.2 is not applicable to higher-speed ethernet which > don't use auto-negotiation, but rather the link training sequence > where the RX side always syncs its clock to the TX side. By "the RX side always syncs its clock to the TX side" do you mean the RX channel synchronizes to the link partner's TX channel? Wow, that brings back the 100 megabit scheme I guess. That's cool, because the same basic idea applies to the PHYTER then. Still we are doing to need a way for user space to query the HW topology to discover whether a given ports may be syntonized from a second port. I don't think your pin selection thing works unless user space can tell what the pins are connected to. Thanks, Richard
> -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: Sunday, August 22, 2021 4:31 AM > Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state > > On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote: > > > Since the 40.5.2 is not applicable to higher-speed ethernet which > > don't use auto-negotiation, but rather the link training sequence > > where the RX side always syncs its clock to the TX side. > > By "the RX side always syncs its clock to the TX side" do you mean the RX > channel synchronizes to the link partner's TX channel? > > Wow, that brings back the 100 megabit scheme I guess. That's cool, because > the same basic idea applies to the PHYTER then. > Yes! Sounds very similar! :) > Still we are doing to need a way for user space to query the HW topology to > discover whether a given ports may be syntonized from a second port. I > don't think your pin selection thing works unless user space can tell what the > pins are connected to. > > Thanks, > Richard And a good catch! I'll update the RFC to add the query functionality and move the SyncE logic/pins to the netdev subsystem. Thanks
> -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: Monday, August 30, 2021 11:06 PM > To: Machnikowski, Maciej <maciej.machnikowski@intel.com> > Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state > > On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote: > > > So to be able to control SyncE we need 2 interfaces: > > - Interface to enable the recovered clock output at the given pin > > - interface to monitor the DPLL to see if the clock that we got is valid, or > not. > > > > If it comes to ESMC (G.8264) messages, SyncE itself can run in 2 modes > (slides 29/30 will give you more details): > > - QL-Disabled - with no ESMC messages - it base on the local information > from the PLL to make all decisions > > - QL-Enabled - that adds ESMC and quality message transfer between the > nodes. > > How do you get the QL codes from this? > > +enum if_eec_state { > + IF_EEC_STATE_INVALID = 0, > + IF_EEC_STATE_FREERUN, > + IF_EEC_STATE_LOCKACQ, > + IF_EEC_STATE_LOCKREC, > + IF_EEC_STATE_LOCKED, > + IF_EEC_STATE_HOLDOVER, > + IF_EEC_STATE_OPEN_LOOP, > + __IF_EEC_STATE_MAX, > +}; This structure is for monitoring the lock state - or in other words - quality of incoming sync signal. The Locked state here means that the frequency used for transmitting the data is syntonized with the input one. If something goes wrong, like the frequency you recover from the link goes beyond the specified range or the external signal is lost, the QL level will change accordingly. The application layer running on top of this API needs to get the proper QL level from the config file (just like the clockClass in PTP) and broadcast it when the state is locked and switch to QL-DNU when you get out of the lock state and expire preset hw-dependent holdover clock. Also, if you are syntonizing to the SyncE clock you need to wait with passing along QL-levels until the state reported by the EEC changes to LOCKED. Regards Maciek
diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c index af3bc65c4595..32b2713f18a5 100644 --- a/drivers/ptp/ptp_chardev.c +++ b/drivers/ptp/ptp_chardev.c @@ -106,6 +106,14 @@ int ptp_open(struct posix_clock *pc, fmode_t fmode) return 0; } +int ptp_get_dpll_state(struct ptp_clock *ptp, struct ptp_dpll_state *ds) +{ + if (!ptp->info->get_dpll_state) + return -EOPNOTSUPP; + + return ptp->info->get_dpll_state(ptp->info, ds); +} + long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg) { struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock); @@ -119,6 +127,7 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg) struct ptp_clock_caps caps; struct ptp_clock_time *pct; unsigned int i, pin_index; + struct ptp_dpll_state ds; struct ptp_pin_desc pd; struct timespec64 ts; int enable, err = 0; @@ -418,6 +427,12 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg) mutex_unlock(&ptp->pincfg_mux); break; + case PTP_DPLL_GETSTATE: + err = ptp_get_dpll_state(ptp, &ds); + if (!err && copy_to_user((void __user *)arg, &ds, sizeof(ds))) + err = -EFAULT; + break; + default: err = -ENOTTY; break; diff --git a/drivers/ptp/ptp_clockmatrix.h b/drivers/ptp/ptp_clockmatrix.h index fb323271063e..0ce2f280c6d3 100644 --- a/drivers/ptp/ptp_clockmatrix.h +++ b/drivers/ptp/ptp_clockmatrix.h @@ -107,18 +107,6 @@ enum scsr_tod_write_type_sel { SCSR_TOD_WR_TYPE_SEL_MAX = SCSR_TOD_WR_TYPE_SEL_DELTA_MINUS, }; -/* Values STATUS.DPLL_SYS_STATUS.DPLL_SYS_STATE */ -enum dpll_state { - DPLL_STATE_MIN = 0, - DPLL_STATE_FREERUN = DPLL_STATE_MIN, - DPLL_STATE_LOCKACQ = 1, - DPLL_STATE_LOCKREC = 2, - DPLL_STATE_LOCKED = 3, - DPLL_STATE_HOLDOVER = 4, - DPLL_STATE_OPEN_LOOP = 5, - DPLL_STATE_MAX = DPLL_STATE_OPEN_LOOP, -}; - struct idtcm; struct idtcm_channel { diff --git a/drivers/ptp/ptp_private.h b/drivers/ptp/ptp_private.h index dba6be477067..c57fb54e2b57 100644 --- a/drivers/ptp/ptp_private.h +++ b/drivers/ptp/ptp_private.h @@ -117,6 +117,8 @@ ssize_t ptp_read(struct posix_clock *pc, __poll_t ptp_poll(struct posix_clock *pc, struct file *fp, poll_table *wait); +int ptp_get_dpll_state(struct ptp_clock *ptp, struct ptp_dpll_state *ds); + /* * see ptp_sysfs.c */ diff --git a/drivers/ptp/ptp_sysfs.c b/drivers/ptp/ptp_sysfs.c index b3d96b747292..fb0890fab266 100644 --- a/drivers/ptp/ptp_sysfs.c +++ b/drivers/ptp/ptp_sysfs.c @@ -302,6 +302,52 @@ static ssize_t max_vclocks_store(struct device *dev, } static DEVICE_ATTR_RW(max_vclocks); +static inline int dpll_state_to_str(enum dpll_state ds, const char **ds_str) +{ + const char * const dpll_state_string[] = { + "FREERUN", + "LOCKACQ", + "LOCKREC", + "LOCKED", + "HOLDOVER", + "OPEN_LOOP", + "INVALID", + }; + size_t max = sizeof(dpll_state_string) / + sizeof(dpll_state_string[0]); + + if (ds < 0 || ds >= max) + return -EINVAL; + *ds_str = dpll_state_string[ds]; + + return 0; +} + +static ssize_t dpll_state_show(struct device *dev, + struct device_attribute *attr, char *page) +{ + struct ptp_clock *ptp = dev_get_drvdata(dev); + struct ptp_dpll_state ds; + const char *ds_str; + ssize_t size = 0; + int i, err; + + err = ptp_get_dpll_state(ptp, &ds); + if (err) + return err; + + for (i = 0; i < ds.dpll_num; i++) { + err = dpll_state_to_str(ds.state[i], &ds_str); + if (err) + return err; + size += snprintf(page + size, PAGE_SIZE - 1, "%d %s\n", + i, ds_str); + } + + return size; +} +static DEVICE_ATTR_RO(dpll_state); + static struct attribute *ptp_attrs[] = { &dev_attr_clock_name.attr, @@ -318,6 +364,8 @@ static struct attribute *ptp_attrs[] = { &dev_attr_pps_enable.attr, &dev_attr_n_vclocks.attr, &dev_attr_max_vclocks.attr, + + &dev_attr_dpll_state.attr, NULL }; diff --git a/include/linux/ptp_clock_kernel.h b/include/linux/ptp_clock_kernel.h index 71fac9237725..d56cd02d778e 100644 --- a/include/linux/ptp_clock_kernel.h +++ b/include/linux/ptp_clock_kernel.h @@ -129,6 +129,13 @@ struct ptp_system_timestamp { * scheduling time (>=0) or negative value in case further * scheduling is not required. * + * @get_dpll_state: Request driver to check and update state of its DPLLs + * (Digital Phase Locked Loop). + * Driver returns structure filled with number of + * available DPLLs and their states. + * On success function returns 0, or negative on failed + * attempt. + * * Drivers should embed their ptp_clock_info within a private * structure, obtaining a reference to it using container_of(). * @@ -160,6 +167,8 @@ struct ptp_clock_info { int (*verify)(struct ptp_clock_info *ptp, unsigned int pin, enum ptp_pin_function func, unsigned int chan); long (*do_aux_work)(struct ptp_clock_info *ptp); + int (*get_dpll_state)(struct ptp_clock_info *ptp, + struct ptp_dpll_state *ds); }; struct ptp_clock; diff --git a/include/uapi/linux/ptp_clock.h b/include/uapi/linux/ptp_clock.h index 1d108d597f66..773505ad59e1 100644 --- a/include/uapi/linux/ptp_clock.h +++ b/include/uapi/linux/ptp_clock.h @@ -198,6 +198,32 @@ struct ptp_pin_desc { unsigned int rsv[5]; }; +enum dpll_state { + DPLL_STATE_MIN = 0, + DPLL_STATE_FREERUN = DPLL_STATE_MIN, + DPLL_STATE_LOCKACQ = 1, + DPLL_STATE_LOCKREC = 2, + DPLL_STATE_LOCKED = 3, + DPLL_STATE_HOLDOVER = 4, + DPLL_STATE_OPEN_LOOP = 5, + DPLL_STATE_MAX = DPLL_STATE_OPEN_LOOP, +}; + +#define PTP_MAX_DPLL_NUM_PER_DEVICE 8 + +struct ptp_dpll_state { + /* + * Number of available dplls on the device. + */ + int dpll_num; + /* + * State of DPLLs. Values defined in enum dpll_states. + * Indexed by DPLL index on the device. + * Valid indicies < dpll_num + */ + __u8 state[PTP_MAX_DPLL_NUM_PER_DEVICE]; +}; + #define PTP_CLK_MAGIC '=' #define PTP_CLOCK_GETCAPS _IOR(PTP_CLK_MAGIC, 1, struct ptp_clock_caps) @@ -223,6 +249,7 @@ struct ptp_pin_desc { _IOWR(PTP_CLK_MAGIC, 17, struct ptp_sys_offset_precise) #define PTP_SYS_OFFSET_EXTENDED2 \ _IOWR(PTP_CLK_MAGIC, 18, struct ptp_sys_offset_extended) +#define PTP_DPLL_GETSTATE _IOR(PTP_CLK_MAGIC, 19, struct ptp_dpll_state) struct ptp_extts_event { struct ptp_clock_time t; /* Time event occured. */
Previously there was no common interface for monitoring synchronization state of Digital Phase Locked Loop. Add interface through PTP ioctl subsystem for tools, as well as sysfs human-friendly part of the interface. enum dpll_state moved to uapi definition, it is required to have common definition of DPLL states in uapi. Add new callback function, must be implemented by ptp enabled driver in order to get the state of dpll. Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com> --- drivers/ptp/ptp_chardev.c | 15 ++++++++++ drivers/ptp/ptp_clockmatrix.h | 12 -------- drivers/ptp/ptp_private.h | 2 ++ drivers/ptp/ptp_sysfs.c | 48 ++++++++++++++++++++++++++++++++ include/linux/ptp_clock_kernel.h | 9 ++++++ include/uapi/linux/ptp_clock.h | 27 ++++++++++++++++++ 6 files changed, 101 insertions(+), 12 deletions(-)