Scheduler atomic and ordered definitions

Improved atomic and ordered synchronisation definitions. Added
order skip function prototype.

Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>

---
This is the ordered queue definition (in patch format) promised
in the call yesterday.
---
 platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
 platform/linux-generic/include/api/odp_schedule.h | 45 ++++++++++++++++++-----
 2 files changed, 64 insertions(+), 12 deletions(-)

"+ * The original enqueue order of the source queue is maintained when
buffers are
+ * enqueued to their destination queue(s) before another schedule call"

Is this assuming that the order will be restored always at the next
enqueue? I think there should be an option to explicitly indicate if the
next enqueue is supposed to restore the order or not, especially when
packets move from queue to queue. Ordered queues are costly compared with
the ordinary ones.

Alex

On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
wrote:

> Improved atomic and ordered synchronisation definitions. Added
> order skip function prototype.
>
> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>
> ---
> This is the ordered queue definition (in patch format) promised
> in the call yesterday.
> ---
>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>  platform/linux-generic/include/api/odp_schedule.h | 45
> ++++++++++++++++++-----
>  2 files changed, 64 insertions(+), 12 deletions(-)
>
> diff --git a/platform/linux-generic/include/api/odp_queue.h
> b/platform/linux-generic/include/api/odp_queue.h
> index b8ac4bb..c0c3969 100644
> --- a/platform/linux-generic/include/api/odp_queue.h
> +++ b/platform/linux-generic/include/api/odp_queue.h
> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>  typedef int odp_schedule_sync_t;
>
>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
> +
> +/**
> + * Atomic queue synchronisation
> + *
> + * The scheduler gives buffers from a queue to a single core at a time.
> This
> + * serialises processing of the buffers from the source queue and helps
> user to
> + * avoid SW locking. Another schedule call will implicitely release the
> atomic
> + * synchronisation of the source queue and free the scheduler to give
> buffers
> + * from the queue to other cores.
> + *
> + * User can hint the scheduler to release the atomic synchronisation
> early with
> + * odp_schedule_release_atomic().
> + */
> +#define ODP_SCHED_SYNC_ATOMIC   1
> +
> +/**
> + * Ordered queue synchronisation
> + *
> + * The scheduler may give out buffers to multiple cores for parallel
> processing.
> + * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call.
> Buffers
> + * from the same ordered (source) queue appear in their original order
> when
> + * dequeued from a destination queue. The destination queue type
> (POLL/SCHED) or
> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
> + *
> + * User can command the scheduler to skip ordering of a buffer with
> + * odp_schedule_skip_order().
> + */
> +#define ODP_SCHED_SYNC_ORDERED  2
>
>  /** Default queue synchronisation */
>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
> diff --git a/platform/linux-generic/include/api/odp_schedule.h
> b/platform/linux-generic/include/api/odp_schedule.h
> index 91fec10..2a1a642 100644
> --- a/platform/linux-generic/include/api/odp_schedule.h
> +++ b/platform/linux-generic/include/api/odp_schedule.h
> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   * for a buffer according to the wait parameter setting. Returns
>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>
>  /**
>   * Schedule one buffer
> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * User can exit the schedule loop without first calling
> odp_schedule_pause().
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>
>
>  /**
> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * Like odp_schedule(), but returns multiple buffers from a queue.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * @return Number of buffers outputed (0 ... num)
>   */
> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
> out_buf[],
> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
> out_buf[],
>                        unsigned int num);
>
>  /**
> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>  void odp_schedule_resume(void);
>
>  /**
> - * Release currently hold atomic context
> + * Release the current atomic context
> + *
> + * This calls is valid only when the source queue has ATOMIC
> synchronisation. It
> + * hints the scheduler that the user has completed processing the critical
> + * section that needs atomic synchronisation. After the call, the
> scheduler is
> + * allowed to give next buffer from the same queue to another core.
> + *
> + * Usage of the call may increase parallelism and thus system
> performance, but
> + * carefulness is needed when splitting processing into critical vs.
> + * non-critical sections.
>   */
>  void odp_schedule_release_atomic(void);
>
>  /**
> + * Skip ordering of the buffer
> + *
> + * This calls is valid only when source queue has ORDERED
> synchronisation. It
> + * commands the scheduler to skip ordering of the buffer. Scheduler
> maintains
> + * ordering between two queues: source and destination. The dest parameter
> + * identifies the destination queue of that pair. After the call,
> ordering is
> + * not maintained for the buffer anymore but user still owns the buffer.
> User
> + * can e.g. store it, free it or enqueue it (to the same or another
> queue).
> + *
> + * @param dest    Destination queue
> + * @param buf     Buffer
> + */
> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
> +
> +
> +/**
>   * Number of scheduling priorities
>   *
>   * @return Number of scheduling priorities
> --
> 2.1.1
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>

I can well imagine the step-by-step order preservation to be simpler to
implement (in SW) but it would also seem to limit performance since the
only way to ensure end-to-end order preservation would be if each
intermediate queue from ingress to egress were an ordered queue.  If there
is a parallel queue anywhere in the chain that would break things.

The question is: Is this restriction needed and/or sufficient for ODP v1.0?


On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
alexandru.badicioiu@linaro.org> wrote:

> "+ * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call"
>
> Is this assuming that the order will be restored always at the next
> enqueue? I think there should be an option to explicitly indicate if the
> next enqueue is supposed to restore the order or not, especially when
> packets move from queue to queue. Ordered queues are costly compared with
> the ordinary ones.
>
> Alex
>
> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
> wrote:
>
>> Improved atomic and ordered synchronisation definitions. Added
>> order skip function prototype.
>>
>> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>>
>> ---
>> This is the ordered queue definition (in patch format) promised
>> in the call yesterday.
>> ---
>>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>>  platform/linux-generic/include/api/odp_schedule.h | 45
>> ++++++++++++++++++-----
>>  2 files changed, 64 insertions(+), 12 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/api/odp_queue.h
>> b/platform/linux-generic/include/api/odp_queue.h
>> index b8ac4bb..c0c3969 100644
>> --- a/platform/linux-generic/include/api/odp_queue.h
>> +++ b/platform/linux-generic/include/api/odp_queue.h
>> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>>  typedef int odp_schedule_sync_t;
>>
>>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
>> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
>> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
>> +
>> +/**
>> + * Atomic queue synchronisation
>> + *
>> + * The scheduler gives buffers from a queue to a single core at a time.
>> This
>> + * serialises processing of the buffers from the source queue and helps
>> user to
>> + * avoid SW locking. Another schedule call will implicitely release the
>> atomic
>> + * synchronisation of the source queue and free the scheduler to give
>> buffers
>> + * from the queue to other cores.
>> + *
>> + * User can hint the scheduler to release the atomic synchronisation
>> early with
>> + * odp_schedule_release_atomic().
>> + */
>> +#define ODP_SCHED_SYNC_ATOMIC   1
>> +
>> +/**
>> + * Ordered queue synchronisation
>> + *
>> + * The scheduler may give out buffers to multiple cores for parallel
>> processing.
>> + * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call.
>> Buffers
>> + * from the same ordered (source) queue appear in their original order
>> when
>> + * dequeued from a destination queue. The destination queue type
>> (POLL/SCHED) or
>> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
>> + *
>> + * User can command the scheduler to skip ordering of a buffer with
>> + * odp_schedule_skip_order().
>> + */
>> +#define ODP_SCHED_SYNC_ORDERED  2
>>
>>  /** Default queue synchronisation */
>>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
>> diff --git a/platform/linux-generic/include/api/odp_schedule.h
>> b/platform/linux-generic/include/api/odp_schedule.h
>> index 91fec10..2a1a642 100644
>> --- a/platform/linux-generic/include/api/odp_schedule.h
>> +++ b/platform/linux-generic/include/api/odp_schedule.h
>> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   * for a buffer according to the wait parameter setting. Returns
>>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>>
>>  /**
>>   * Schedule one buffer
>> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * User can exit the schedule loop without first calling
>> odp_schedule_pause().
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>>
>>
>>  /**
>> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * Like odp_schedule(), but returns multiple buffers from a queue.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * @return Number of buffers outputed (0 ... num)
>>   */
>> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
>> out_buf[],
>> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
>> out_buf[],
>>                        unsigned int num);
>>
>>  /**
>> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>>  void odp_schedule_resume(void);
>>
>>  /**
>> - * Release currently hold atomic context
>> + * Release the current atomic context
>> + *
>> + * This calls is valid only when the source queue has ATOMIC
>> synchronisation. It
>> + * hints the scheduler that the user has completed processing the
>> critical
>> + * section that needs atomic synchronisation. After the call, the
>> scheduler is
>> + * allowed to give next buffer from the same queue to another core.
>> + *
>> + * Usage of the call may increase parallelism and thus system
>> performance, but
>> + * carefulness is needed when splitting processing into critical vs.
>> + * non-critical sections.
>>   */
>>  void odp_schedule_release_atomic(void);
>>
>>  /**
>> + * Skip ordering of the buffer
>> + *
>> + * This calls is valid only when source queue has ORDERED
>> synchronisation. It
>> + * commands the scheduler to skip ordering of the buffer. Scheduler
>> maintains
>> + * ordering between two queues: source and destination. The dest
>> parameter
>> + * identifies the destination queue of that pair. After the call,
>> ordering is
>> + * not maintained for the buffer anymore but user still owns the buffer.
>> User
>> + * can e.g. store it, free it or enqueue it (to the same or another
>> queue).
>> + *
>> + * @param dest    Destination queue
>> + * @param buf     Buffer
>> + */
>> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
>> +
>> +
>> +/**
>>   * Number of scheduling priorities
>>   *
>>   * @return Number of scheduling priorities
>> --
>> 2.1.1
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>

This may well be a reasonable restriction for ODP v1.0 but I believe it's
something we need to put on the list for "production grade" improvements
for 2015.

Bill

On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
petri.savolainen@nsn.com> wrote:

>  Yes, it’s step-by-step and I think it’s the level of ordering we need
> for v1.0. Most SoCs can implement it, even when  the HW scheduler would not
> have order support but only atomic/parallel. This way defined atomic
> scheduling can be used to implement functionality correct ordered queues,
> the throughput is not improved but it functions correctly.
>
>
>
> -Petri
>
>
>
>
>
> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
> *Sent:* Friday, October 31, 2014 2:48 PM
> *To:* Alexandru Badicioiu
> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> I can well imagine the step-by-step order preservation to be simpler to
> implement (in SW) but it would also seem to limit performance since the
> only way to ensure end-to-end order preservation would be if each
> intermediate queue from ingress to egress were an ordered queue.  If there
> is a parallel queue anywhere in the chain that would break things.
>
>
>
> The question is: Is this restriction needed and/or sufficient for ODP
> v1.0?
>
>
>
> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
> alexandru.badicioiu@linaro.org> wrote:
>
> "+ * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call"
>
>
>
> Is this assuming that the order will be restored always at the next
> enqueue? I think there should be an option to explicitly indicate if the
> next enqueue is supposed to restore the order or not, especially when
> packets move from queue to queue. Ordered queues are costly compared with
> the ordinary ones.
>
>
>
> Alex
>
>
>
> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
> wrote:
>
> Improved atomic and ordered synchronisation definitions. Added
> order skip function prototype.
>
> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>
> ---
> This is the ordered queue definition (in patch format) promised
> in the call yesterday.
> ---
>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>  platform/linux-generic/include/api/odp_schedule.h | 45
> ++++++++++++++++++-----
>  2 files changed, 64 insertions(+), 12 deletions(-)
>
> diff --git a/platform/linux-generic/include/api/odp_queue.h
> b/platform/linux-generic/include/api/odp_queue.h
> index b8ac4bb..c0c3969 100644
> --- a/platform/linux-generic/include/api/odp_queue.h
> +++ b/platform/linux-generic/include/api/odp_queue.h
> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>  typedef int odp_schedule_sync_t;
>
>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
> +
> +/**
> + * Atomic queue synchronisation
> + *
> + * The scheduler gives buffers from a queue to a single core at a time.
> This
> + * serialises processing of the buffers from the source queue and helps
> user to
> + * avoid SW locking. Another schedule call will implicitely release the
> atomic
> + * synchronisation of the source queue and free the scheduler to give
> buffers
> + * from the queue to other cores.
> + *
> + * User can hint the scheduler to release the atomic synchronisation
> early with
> + * odp_schedule_release_atomic().
> + */
> +#define ODP_SCHED_SYNC_ATOMIC   1
> +
> +/**
> + * Ordered queue synchronisation
> + *
> + * The scheduler may give out buffers to multiple cores for parallel
> processing.
> + * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call.
> Buffers
> + * from the same ordered (source) queue appear in their original order
> when
> + * dequeued from a destination queue. The destination queue type
> (POLL/SCHED) or
> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
> + *
> + * User can command the scheduler to skip ordering of a buffer with
> + * odp_schedule_skip_order().
> + */
> +#define ODP_SCHED_SYNC_ORDERED  2
>
>  /** Default queue synchronisation */
>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
> diff --git a/platform/linux-generic/include/api/odp_schedule.h
> b/platform/linux-generic/include/api/odp_schedule.h
> index 91fec10..2a1a642 100644
> --- a/platform/linux-generic/include/api/odp_schedule.h
> +++ b/platform/linux-generic/include/api/odp_schedule.h
> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   * for a buffer according to the wait parameter setting. Returns
>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>
>  /**
>   * Schedule one buffer
> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * User can exit the schedule loop without first calling
> odp_schedule_pause().
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>
>
>  /**
> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * Like odp_schedule(), but returns multiple buffers from a queue.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * @return Number of buffers outputed (0 ... num)
>   */
> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
> out_buf[],
> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
> out_buf[],
>                        unsigned int num);
>
>  /**
> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>  void odp_schedule_resume(void);
>
>  /**
> - * Release currently hold atomic context
> + * Release the current atomic context
> + *
> + * This calls is valid only when the source queue has ATOMIC
> synchronisation. It
> + * hints the scheduler that the user has completed processing the critical
> + * section that needs atomic synchronisation. After the call, the
> scheduler is
> + * allowed to give next buffer from the same queue to another core.
> + *
> + * Usage of the call may increase parallelism and thus system
> performance, but
> + * carefulness is needed when splitting processing into critical vs.
> + * non-critical sections.
>   */
>  void odp_schedule_release_atomic(void);
>
>  /**
> + * Skip ordering of the buffer
> + *
> + * This calls is valid only when source queue has ORDERED
> synchronisation. It
> + * commands the scheduler to skip ordering of the buffer. Scheduler
> maintains
> + * ordering between two queues: source and destination. The dest parameter
> + * identifies the destination queue of that pair. After the call,
> ordering is
> + * not maintained for the buffer anymore but user still owns the buffer.
> User
> + * can e.g. store it, free it or enqueue it (to the same or another
> queue).
> + *
> + * @param dest    Destination queue
> + * @param buf     Buffer
> + */
> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
> +
> +
> +/**
>   * Number of scheduling priorities
>   *
>   * @return Number of scheduling priorities
> --
> 2.1.1
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>

In this case the "dest" argument in the following function:
+void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
looses it's meaning.

Alex



On 31 October 2014 14:59, Bill Fischofer <bill.fischofer@linaro.org> wrote:

> This may well be a reasonable restriction for ODP v1.0 but I believe it's
> something we need to put on the list for "production grade" improvements
> for 2015.
>
> Bill
>
> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
> petri.savolainen@nsn.com> wrote:
>
>>  Yes, it’s step-by-step and I think it’s the level of ordering we need
>> for v1.0. Most SoCs can implement it, even when  the HW scheduler would not
>> have order support but only atomic/parallel. This way defined atomic
>> scheduling can be used to implement functionality correct ordered queues,
>> the throughput is not improved but it functions correctly.
>>
>>
>>
>> -Petri
>>
>>
>>
>>
>>
>> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
>> *Sent:* Friday, October 31, 2014 2:48 PM
>> *To:* Alexandru Badicioiu
>> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> I can well imagine the step-by-step order preservation to be simpler to
>> implement (in SW) but it would also seem to limit performance since the
>> only way to ensure end-to-end order preservation would be if each
>> intermediate queue from ingress to egress were an ordered queue.  If there
>> is a parallel queue anywhere in the chain that would break things.
>>
>>
>>
>> The question is: Is this restriction needed and/or sufficient for ODP
>> v1.0?
>>
>>
>>
>> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
>> alexandru.badicioiu@linaro.org> wrote:
>>
>> "+ * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call"
>>
>>
>>
>> Is this assuming that the order will be restored always at the next
>> enqueue? I think there should be an option to explicitly indicate if the
>> next enqueue is supposed to restore the order or not, especially when
>> packets move from queue to queue. Ordered queues are costly compared with
>> the ordinary ones.
>>
>>
>>
>> Alex
>>
>>
>>
>> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
>> wrote:
>>
>> Improved atomic and ordered synchronisation definitions. Added
>> order skip function prototype.
>>
>> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>>
>> ---
>> This is the ordered queue definition (in patch format) promised
>> in the call yesterday.
>> ---
>>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>>  platform/linux-generic/include/api/odp_schedule.h | 45
>> ++++++++++++++++++-----
>>  2 files changed, 64 insertions(+), 12 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/api/odp_queue.h
>> b/platform/linux-generic/include/api/odp_queue.h
>> index b8ac4bb..c0c3969 100644
>> --- a/platform/linux-generic/include/api/odp_queue.h
>> +++ b/platform/linux-generic/include/api/odp_queue.h
>> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>>  typedef int odp_schedule_sync_t;
>>
>>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
>> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
>> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
>> +
>> +/**
>> + * Atomic queue synchronisation
>> + *
>> + * The scheduler gives buffers from a queue to a single core at a time.
>> This
>> + * serialises processing of the buffers from the source queue and helps
>> user to
>> + * avoid SW locking. Another schedule call will implicitely release the
>> atomic
>> + * synchronisation of the source queue and free the scheduler to give
>> buffers
>> + * from the queue to other cores.
>> + *
>> + * User can hint the scheduler to release the atomic synchronisation
>> early with
>> + * odp_schedule_release_atomic().
>> + */
>> +#define ODP_SCHED_SYNC_ATOMIC   1
>> +
>> +/**
>> + * Ordered queue synchronisation
>> + *
>> + * The scheduler may give out buffers to multiple cores for parallel
>> processing.
>> + * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call.
>> Buffers
>> + * from the same ordered (source) queue appear in their original order
>> when
>> + * dequeued from a destination queue. The destination queue type
>> (POLL/SCHED) or
>> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
>> + *
>> + * User can command the scheduler to skip ordering of a buffer with
>> + * odp_schedule_skip_order().
>> + */
>> +#define ODP_SCHED_SYNC_ORDERED  2
>>
>>  /** Default queue synchronisation */
>>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
>> diff --git a/platform/linux-generic/include/api/odp_schedule.h
>> b/platform/linux-generic/include/api/odp_schedule.h
>> index 91fec10..2a1a642 100644
>> --- a/platform/linux-generic/include/api/odp_schedule.h
>> +++ b/platform/linux-generic/include/api/odp_schedule.h
>> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   * for a buffer according to the wait parameter setting. Returns
>>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>>
>>  /**
>>   * Schedule one buffer
>> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * User can exit the schedule loop without first calling
>> odp_schedule_pause().
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>>
>>
>>  /**
>> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * Like odp_schedule(), but returns multiple buffers from a queue.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * @return Number of buffers outputed (0 ... num)
>>   */
>> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
>> out_buf[],
>> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
>> out_buf[],
>>                        unsigned int num);
>>
>>  /**
>> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>>  void odp_schedule_resume(void);
>>
>>  /**
>> - * Release currently hold atomic context
>> + * Release the current atomic context
>> + *
>> + * This calls is valid only when the source queue has ATOMIC
>> synchronisation. It
>> + * hints the scheduler that the user has completed processing the
>> critical
>> + * section that needs atomic synchronisation. After the call, the
>> scheduler is
>> + * allowed to give next buffer from the same queue to another core.
>> + *
>> + * Usage of the call may increase parallelism and thus system
>> performance, but
>> + * carefulness is needed when splitting processing into critical vs.
>> + * non-critical sections.
>>   */
>>  void odp_schedule_release_atomic(void);
>>
>>  /**
>> + * Skip ordering of the buffer
>> + *
>> + * This calls is valid only when source queue has ORDERED
>> synchronisation. It
>> + * commands the scheduler to skip ordering of the buffer. Scheduler
>> maintains
>> + * ordering between two queues: source and destination. The dest
>> parameter
>> + * identifies the destination queue of that pair. After the call,
>> ordering is
>> + * not maintained for the buffer anymore but user still owns the buffer.
>> User
>> + * can e.g. store it, free it or enqueue it (to the same or another
>> queue).
>> + *
>> + * @param dest    Destination queue
>> + * @param buf     Buffer
>> + */
>> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
>> +
>> +
>> +/**
>>   * Number of scheduling priorities
>>   *
>>   * @return Number of scheduling priorities
>> --
>> 2.1.1
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>
>

It would be documentation in v1.0 but worth keeping since if the
step-by-step restriction is relaxed next year it would mean we wouldn't
have to change the API.

On Fri, Oct 31, 2014 at 8:10 AM, Alexandru Badicioiu <
alexandru.badicioiu@linaro.org> wrote:

> In this case the "dest" argument in the following function:
> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
> looses it's meaning.
>
> Alex
>
>
>
> On 31 October 2014 14:59, Bill Fischofer <bill.fischofer@linaro.org>
> wrote:
>
>> This may well be a reasonable restriction for ODP v1.0 but I believe it's
>> something we need to put on the list for "production grade" improvements
>> for 2015.
>>
>> Bill
>>
>> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
>> petri.savolainen@nsn.com> wrote:
>>
>>>  Yes, it’s step-by-step and I think it’s the level of ordering we need
>>> for v1.0. Most SoCs can implement it, even when  the HW scheduler would not
>>> have order support but only atomic/parallel. This way defined atomic
>>> scheduling can be used to implement functionality correct ordered queues,
>>> the throughput is not improved but it functions correctly.
>>>
>>>
>>>
>>> -Petri
>>>
>>>
>>>
>>>
>>>
>>> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
>>> *Sent:* Friday, October 31, 2014 2:48 PM
>>> *To:* Alexandru Badicioiu
>>> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
>>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered
>>> definitions
>>>
>>>
>>>
>>> I can well imagine the step-by-step order preservation to be simpler to
>>> implement (in SW) but it would also seem to limit performance since the
>>> only way to ensure end-to-end order preservation would be if each
>>> intermediate queue from ingress to egress were an ordered queue.  If there
>>> is a parallel queue anywhere in the chain that would break things.
>>>
>>>
>>>
>>> The question is: Is this restriction needed and/or sufficient for ODP
>>> v1.0?
>>>
>>>
>>>
>>> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
>>> alexandru.badicioiu@linaro.org> wrote:
>>>
>>> "+ * The original enqueue order of the source queue is maintained when
>>> buffers are
>>> + * enqueued to their destination queue(s) before another schedule call"
>>>
>>>
>>>
>>> Is this assuming that the order will be restored always at the next
>>> enqueue? I think there should be an option to explicitly indicate if the
>>> next enqueue is supposed to restore the order or not, especially when
>>> packets move from queue to queue. Ordered queues are costly compared with
>>> the ordinary ones.
>>>
>>>
>>>
>>> Alex
>>>
>>>
>>>
>>> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
>>> wrote:
>>>
>>> Improved atomic and ordered synchronisation definitions. Added
>>> order skip function prototype.
>>>
>>> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>>>
>>> ---
>>> This is the ordered queue definition (in patch format) promised
>>> in the call yesterday.
>>> ---
>>>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>>>  platform/linux-generic/include/api/odp_schedule.h | 45
>>> ++++++++++++++++++-----
>>>  2 files changed, 64 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/platform/linux-generic/include/api/odp_queue.h
>>> b/platform/linux-generic/include/api/odp_queue.h
>>> index b8ac4bb..c0c3969 100644
>>> --- a/platform/linux-generic/include/api/odp_queue.h
>>> +++ b/platform/linux-generic/include/api/odp_queue.h
>>> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>>>  typedef int odp_schedule_sync_t;
>>>
>>>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
>>> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
>>> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
>>> +
>>> +/**
>>> + * Atomic queue synchronisation
>>> + *
>>> + * The scheduler gives buffers from a queue to a single core at a time.
>>> This
>>> + * serialises processing of the buffers from the source queue and helps
>>> user to
>>> + * avoid SW locking. Another schedule call will implicitely release the
>>> atomic
>>> + * synchronisation of the source queue and free the scheduler to give
>>> buffers
>>> + * from the queue to other cores.
>>> + *
>>> + * User can hint the scheduler to release the atomic synchronisation
>>> early with
>>> + * odp_schedule_release_atomic().
>>> + */
>>> +#define ODP_SCHED_SYNC_ATOMIC   1
>>> +
>>> +/**
>>> + * Ordered queue synchronisation
>>> + *
>>> + * The scheduler may give out buffers to multiple cores for parallel
>>> processing.
>>> + * The original enqueue order of the source queue is maintained when
>>> buffers are
>>> + * enqueued to their destination queue(s) before another schedule call.
>>> Buffers
>>> + * from the same ordered (source) queue appear in their original order
>>> when
>>> + * dequeued from a destination queue. The destination queue type
>>> (POLL/SCHED) or
>>> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
>>> + *
>>> + * User can command the scheduler to skip ordering of a buffer with
>>> + * odp_schedule_skip_order().
>>> + */
>>> +#define ODP_SCHED_SYNC_ORDERED  2
>>>
>>>  /** Default queue synchronisation */
>>>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
>>> diff --git a/platform/linux-generic/include/api/odp_schedule.h
>>> b/platform/linux-generic/include/api/odp_schedule.h
>>> index 91fec10..2a1a642 100644
>>> --- a/platform/linux-generic/include/api/odp_schedule.h
>>> +++ b/platform/linux-generic/include/api/odp_schedule.h
>>> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>>   * for a buffer according to the wait parameter setting. Returns
>>>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>>>   *
>>> - * @param from    Output parameter for the source queue (where the
>>> buffer was
>>> - *                dequeued from). Ignored if NULL.
>>> + * @param src     The source queue (output). Indicates from which queue
>>> the
>>> + *                buffer was dequeued. Ignored if NULL.
>>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>>> if set to
>>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>>> ODP_SCHED_NO_WAIT.
>>>   *                Use odp_schedule_wait_time() to convert time to other
>>> wait
>>> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>>   *
>>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>>   */
>>> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
>>> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>>>
>>>  /**
>>>   * Schedule one buffer
>>> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>>> wait);
>>>   *
>>>   * User can exit the schedule loop without first calling
>>> odp_schedule_pause().
>>>   *
>>> - * @param from    Output parameter for the source queue (where the
>>> buffer was
>>> - *                dequeued from). Ignored if NULL.
>>> + * @param src     The source queue (output). Indicates from which queue
>>> the
>>> + *                buffer was dequeued. Ignored if NULL.
>>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>>> if set to
>>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>>> ODP_SCHED_NO_WAIT.
>>>   *                Use odp_schedule_wait_time() to convert time to other
>>> wait
>>> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>>> wait);
>>>   *
>>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>>   */
>>> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
>>> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>>>
>>>
>>>  /**
>>> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>>> uint64_t wait);
>>>   *
>>>   * Like odp_schedule(), but returns multiple buffers from a queue.
>>>   *
>>> - * @param from    Output parameter for the source queue (where the
>>> buffer was
>>> - *                dequeued from). Ignored if NULL.
>>> + * @param src     The source queue (output). Indicates from which queue
>>> the
>>> + *                buffer was dequeued. Ignored if NULL.
>>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>>> if set to
>>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>>> ODP_SCHED_NO_WAIT.
>>>   *                Use odp_schedule_wait_time() to convert time to other
>>> wait
>>> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>>> uint64_t wait);
>>>   *
>>>   * @return Number of buffers outputed (0 ... num)
>>>   */
>>> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
>>> out_buf[],
>>> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
>>> out_buf[],
>>>                        unsigned int num);
>>>
>>>  /**
>>> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>>>  void odp_schedule_resume(void);
>>>
>>>  /**
>>> - * Release currently hold atomic context
>>> + * Release the current atomic context
>>> + *
>>> + * This calls is valid only when the source queue has ATOMIC
>>> synchronisation. It
>>> + * hints the scheduler that the user has completed processing the
>>> critical
>>> + * section that needs atomic synchronisation. After the call, the
>>> scheduler is
>>> + * allowed to give next buffer from the same queue to another core.
>>> + *
>>> + * Usage of the call may increase parallelism and thus system
>>> performance, but
>>> + * carefulness is needed when splitting processing into critical vs.
>>> + * non-critical sections.
>>>   */
>>>  void odp_schedule_release_atomic(void);
>>>
>>>  /**
>>> + * Skip ordering of the buffer
>>> + *
>>> + * This calls is valid only when source queue has ORDERED
>>> synchronisation. It
>>> + * commands the scheduler to skip ordering of the buffer. Scheduler
>>> maintains
>>> + * ordering between two queues: source and destination. The dest
>>> parameter
>>> + * identifies the destination queue of that pair. After the call,
>>> ordering is
>>> + * not maintained for the buffer anymore but user still owns the
>>> buffer. User
>>> + * can e.g. store it, free it or enqueue it (to the same or another
>>> queue).
>>> + *
>>> + * @param dest    Destination queue
>>> + * @param buf     Buffer
>>> + */
>>> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
>>> +
>>> +
>>> +/**
>>>   * Number of scheduling priorities
>>>   *
>>>   * @return Number of scheduling priorities
>>> --
>>> 2.1.1
>>>
>>>
>>> _______________________________________________
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>>
>>>
>>
>>
>

Thanks Gilad.  Can you elaborate on that a bit more?  I can understand how
step-by-step would be potentially easier to implement, but does it also
capture the majority of expected application use cases?

A good confirmation that this is the correct approach for v1.0 though.

On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef <giladb@ezchip.com> wrote:

>
>
> For what it's worth, as a SoC vendor rep. that has ordered queue in HW, Petri's
> definition is actually preferred for us even going forward J
>
>
>
>
>
> Thanks,
>
> Gilad
>
>
>
> *Gilad Ben-Yossef*
>
> Software Architect
>
> EZchip Technologies Ltd.
>
> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>
> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>
> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>
>
>
> *"Ethernet always wins."*
>
>         — Andy Bechtolsheim
>
>
>
> *From:* lng-odp-bounces@lists.linaro.org [mailto:
> lng-odp-bounces@lists.linaro.org] *On Behalf Of *Bill Fischofer
> *Sent:* Friday, October 31, 2014 2:59 PM
> *To:* Savolainen, Petri (NSN - FI/Espoo)
> *Cc:* lng-odp@lists.linaro.org
>
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> This may well be a reasonable restriction for ODP v1.0 but I believe it's
> something we need to put on the list for "production grade" improvements
> for 2015.
>
>
>
> Bill
>
>
>
> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
> petri.savolainen@nsn.com> wrote:
>
> Yes, it’s step-by-step and I think it’s the level of ordering we need for
> v1.0. Most SoCs can implement it, even when  the HW scheduler would not
> have order support but only atomic/parallel. This way defined atomic
> scheduling can be used to implement functionality correct ordered queues,
> the throughput is not improved but it functions correctly.
>
>
>
> -Petri
>
>
>
>
>
> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
> *Sent:* Friday, October 31, 2014 2:48 PM
> *To:* Alexandru Badicioiu
> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> I can well imagine the step-by-step order preservation to be simpler to
> implement (in SW) but it would also seem to limit performance since the
> only way to ensure end-to-end order preservation would be if each
> intermediate queue from ingress to egress were an ordered queue.  If there
> is a parallel queue anywhere in the chain that would break things.
>
>
>
> The question is: Is this restriction needed and/or sufficient for ODP
> v1.0?
>
>
>
> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
> alexandru.badicioiu@linaro.org> wrote:
>
> "+ * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call"
>
>
>
> Is this assuming that the order will be restored always at the next
> enqueue? I think there should be an option to explicitly indicate if the
> next enqueue is supposed to restore the order or not, especially when
> packets move from queue to queue. Ordered queues are costly compared with
> the ordinary ones.
>
>
>
> Alex
>
>
>
> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
> wrote:
>
> Improved atomic and ordered synchronisation definitions. Added
> order skip function prototype.
>
> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>
> ---
> This is the ordered queue definition (in patch format) promised
> in the call yesterday.
> ---
>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>  platform/linux-generic/include/api/odp_schedule.h | 45
> ++++++++++++++++++-----
>  2 files changed, 64 insertions(+), 12 deletions(-)
>
> diff --git a/platform/linux-generic/include/api/odp_queue.h
> b/platform/linux-generic/include/api/odp_queue.h
> index b8ac4bb..c0c3969 100644
> --- a/platform/linux-generic/include/api/odp_queue.h
> +++ b/platform/linux-generic/include/api/odp_queue.h
> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>  typedef int odp_schedule_sync_t;
>
>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
> +
> +/**
> + * Atomic queue synchronisation
> + *
> + * The scheduler gives buffers from a queue to a single core at a time.
> This
> + * serialises processing of the buffers from the source queue and helps
> user to
> + * avoid SW locking. Another schedule call will implicitely release the
> atomic
> + * synchronisation of the source queue and free the scheduler to give
> buffers
> + * from the queue to other cores.
> + *
> + * User can hint the scheduler to release the atomic synchronisation
> early with
> + * odp_schedule_release_atomic().
> + */
> +#define ODP_SCHED_SYNC_ATOMIC   1
> +
> +/**
> + * Ordered queue synchronisation
> + *
> + * The scheduler may give out buffers to multiple cores for parallel
> processing.
> + * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call.
> Buffers
> + * from the same ordered (source) queue appear in their original order
> when
> + * dequeued from a destination queue. The destination queue type
> (POLL/SCHED) or
> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
> + *
> + * User can command the scheduler to skip ordering of a buffer with
> + * odp_schedule_skip_order().
> + */
> +#define ODP_SCHED_SYNC_ORDERED  2
>
>  /** Default queue synchronisation */
>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
> diff --git a/platform/linux-generic/include/api/odp_schedule.h
> b/platform/linux-generic/include/api/odp_schedule.h
> index 91fec10..2a1a642 100644
> --- a/platform/linux-generic/include/api/odp_schedule.h
> +++ b/platform/linux-generic/include/api/odp_schedule.h
> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   * for a buffer according to the wait parameter setting. Returns
>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>
>  /**
>   * Schedule one buffer
> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * User can exit the schedule loop without first calling
> odp_schedule_pause().
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>
>
>  /**
> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * Like odp_schedule(), but returns multiple buffers from a queue.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * @return Number of buffers outputed (0 ... num)
>   */
> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
> out_buf[],
> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
> out_buf[],
>                        unsigned int num);
>
>  /**
> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>  void odp_schedule_resume(void);
>
>  /**
> - * Release currently hold atomic context
> + * Release the current atomic context
> + *
> + * This calls is valid only when the source queue has ATOMIC
> synchronisation. It
> + * hints the scheduler that the user has completed processing the critical
> + * section that needs atomic synchronisation. After the call, the
> scheduler is
> + * allowed to give next buffer from the same queue to another core.
> + *
> + * Usage of the call may increase parallelism and thus system
> performance, but
> + * carefulness is needed when splitting processing into critical vs.
> + * non-critical sections.
>   */
>  void odp_schedule_release_atomic(void);
>
>  /**
> + * Skip ordering of the buffer
> + *
> + * This calls is valid only when source queue has ORDERED
> synchronisation. It
> + * commands the scheduler to skip ordering of the buffer. Scheduler
> maintains
> + * ordering between two queues: source and destination. The dest parameter
> + * identifies the destination queue of that pair. After the call,
> ordering is
> + * not maintained for the buffer anymore but user still owns the buffer.
> User
> + * can e.g. store it, free it or enqueue it (to the same or another
> queue).
> + *
> + * @param dest    Destination queue
> + * @param buf     Buffer
> + */
> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
> +
> +
> +/**
>   * Number of scheduling priorities
>   *
>   * @return Number of scheduling priorities
> --
> 2.1.1
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
>

OK, thanks.  That certainly matches the step-by-step semantics Petri
proposed.  We'll have to discuss what, if anything, more we need regarding
ORDERED queues post-v1.0 next year.

Bill

On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef <giladb@ezchip.com> wrote:

>
>
>
>
> What NPS does in HW is that each buffer you dequeue actually stays on the
> queue (but is marked as consumed by the application) and you are given a
> "tag" for it.
>
>
>
> When I want to queue the buffer to the next, I give the HW the tag I got,
> not the buffer handle and what the HW does is only than fully dequeue the
> buffer from the original queue before queuing it into the new queue – but
> only after all preceding (older) buffers on the original queue have been
> fully dequeued,
>
>
>
> Free of a buffer causes a dequeue from the original queue as well.
>
>
>
> And I can also give release a buffer from the original queue (give the HW
> the tag back and say "you don't have to wait for this buffer any longer).
>
>
>
> I hope this helps,
>
> Gilad
>
>
>
> *Gilad Ben-Yossef*
>
> Software Architect
>
> EZchip Technologies Ltd.
>
> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>
> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>
> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>
>
>
> *"Ethernet always wins."*
>
>         — Andy Bechtolsheim
>
>
>
> *From:* Bill Fischofer [mailto:bill.fischofer@linaro.org]
> *Sent:* Sunday, November 02, 2014 5:44 PM
> *To:* Gilad Ben Yossef
> *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org
>
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> Thanks Gilad.  Can you elaborate on that a bit more?  I can understand how
> step-by-step would be potentially easier to implement, but does it also
> capture the majority of expected application use cases?
>
>
>
> A good confirmation that this is the correct approach for v1.0 though.
>
>
>
> On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef <giladb@ezchip.com>
> wrote:
>
>
>
> For what it's worth, as a SoC vendor rep. that has ordered queue in HW,
> Petri's definition is actually preferred for us even going forward J
>
>
>
>
>
> Thanks,
>
> Gilad
>
>
>
> *Gilad Ben-Yossef*
>
> Software Architect
>
> EZchip Technologies Ltd.
>
> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>
> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>
> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>
>
>
> *"Ethernet always wins."*
>
>         — Andy Bechtolsheim
>
>
>
> *From:* lng-odp-bounces@lists.linaro.org [mailto:
> lng-odp-bounces@lists.linaro.org] *On Behalf Of *Bill Fischofer
> *Sent:* Friday, October 31, 2014 2:59 PM
> *To:* Savolainen, Petri (NSN - FI/Espoo)
> *Cc:* lng-odp@lists.linaro.org
>
>
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> This may well be a reasonable restriction for ODP v1.0 but I believe it's
> something we need to put on the list for "production grade" improvements
> for 2015.
>
>
>
> Bill
>
>
>
> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
> petri.savolainen@nsn.com> wrote:
>
> Yes, it’s step-by-step and I think it’s the level of ordering we need for
> v1.0. Most SoCs can implement it, even when  the HW scheduler would not
> have order support but only atomic/parallel. This way defined atomic
> scheduling can be used to implement functionality correct ordered queues,
> the throughput is not improved but it functions correctly.
>
>
>
> -Petri
>
>
>
>
>
> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
> *Sent:* Friday, October 31, 2014 2:48 PM
> *To:* Alexandru Badicioiu
> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> I can well imagine the step-by-step order preservation to be simpler to
> implement (in SW) but it would also seem to limit performance since the
> only way to ensure end-to-end order preservation would be if each
> intermediate queue from ingress to egress were an ordered queue.  If there
> is a parallel queue anywhere in the chain that would break things.
>
>
>
> The question is: Is this restriction needed and/or sufficient for ODP
> v1.0?
>
>
>
> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
> alexandru.badicioiu@linaro.org> wrote:
>
> "+ * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call"
>
>
>
> Is this assuming that the order will be restored always at the next
> enqueue? I think there should be an option to explicitly indicate if the
> next enqueue is supposed to restore the order or not, especially when
> packets move from queue to queue. Ordered queues are costly compared with
> the ordinary ones.
>
>
>
> Alex
>
>
>
> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
> wrote:
>
> Improved atomic and ordered synchronisation definitions. Added
> order skip function prototype.
>
> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>
> ---
> This is the ordered queue definition (in patch format) promised
> in the call yesterday.
> ---
>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>  platform/linux-generic/include/api/odp_schedule.h | 45
> ++++++++++++++++++-----
>  2 files changed, 64 insertions(+), 12 deletions(-)
>
> diff --git a/platform/linux-generic/include/api/odp_queue.h
> b/platform/linux-generic/include/api/odp_queue.h
> index b8ac4bb..c0c3969 100644
> --- a/platform/linux-generic/include/api/odp_queue.h
> +++ b/platform/linux-generic/include/api/odp_queue.h
> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>  typedef int odp_schedule_sync_t;
>
>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
> +
> +/**
> + * Atomic queue synchronisation
> + *
> + * The scheduler gives buffers from a queue to a single core at a time.
> This
> + * serialises processing of the buffers from the source queue and helps
> user to
> + * avoid SW locking. Another schedule call will implicitely release the
> atomic
> + * synchronisation of the source queue and free the scheduler to give
> buffers
> + * from the queue to other cores.
> + *
> + * User can hint the scheduler to release the atomic synchronisation
> early with
> + * odp_schedule_release_atomic().
> + */
> +#define ODP_SCHED_SYNC_ATOMIC   1
> +
> +/**
> + * Ordered queue synchronisation
> + *
> + * The scheduler may give out buffers to multiple cores for parallel
> processing.
> + * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call.
> Buffers
> + * from the same ordered (source) queue appear in their original order
> when
> + * dequeued from a destination queue. The destination queue type
> (POLL/SCHED) or
> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
> + *
> + * User can command the scheduler to skip ordering of a buffer with
> + * odp_schedule_skip_order().
> + */
> +#define ODP_SCHED_SYNC_ORDERED  2
>
>  /** Default queue synchronisation */
>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
> diff --git a/platform/linux-generic/include/api/odp_schedule.h
> b/platform/linux-generic/include/api/odp_schedule.h
> index 91fec10..2a1a642 100644
> --- a/platform/linux-generic/include/api/odp_schedule.h
> +++ b/platform/linux-generic/include/api/odp_schedule.h
> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   * for a buffer according to the wait parameter setting. Returns
>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>
>  /**
>   * Schedule one buffer
> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * User can exit the schedule loop without first calling
> odp_schedule_pause().
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>
>
>  /**
> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * Like odp_schedule(), but returns multiple buffers from a queue.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * @return Number of buffers outputed (0 ... num)
>   */
> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
> out_buf[],
> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
> out_buf[],
>                        unsigned int num);
>
>  /**
> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>  void odp_schedule_resume(void);
>
>  /**
> - * Release currently hold atomic context
> + * Release the current atomic context
> + *
> + * This calls is valid only when the source queue has ATOMIC
> synchronisation. It
> + * hints the scheduler that the user has completed processing the critical
> + * section that needs atomic synchronisation. After the call, the
> scheduler is
> + * allowed to give next buffer from the same queue to another core.
> + *
> + * Usage of the call may increase parallelism and thus system
> performance, but
> + * carefulness is needed when splitting processing into critical vs.
> + * non-critical sections.
>   */
>  void odp_schedule_release_atomic(void);
>
>  /**
> + * Skip ordering of the buffer
> + *
> + * This calls is valid only when source queue has ORDERED
> synchronisation. It
> + * commands the scheduler to skip ordering of the buffer. Scheduler
> maintains
> + * ordering between two queues: source and destination. The dest parameter
> + * identifies the destination queue of that pair. After the call,
> ordering is
> + * not maintained for the buffer anymore but user still owns the buffer.
> User
> + * can e.g. store it, free it or enqueue it (to the same or another
> queue).
> + *
> + * @param dest    Destination queue
> + * @param buf     Buffer
> + */
> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
> +
> +
> +/**
>   * Number of scheduling priorities
>   *
>   * @return Number of scheduling priorities
> --
> 2.1.1
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
>
>
>

In addition to skip order function I think we need to handle the case when
a packet gets fragmented (e.g. IP fragmentation). In this case the
fragments should share the original sequence number and the order
restoration logic should be aware when the last fragment occurs.

Alex

On 3 November 2014 13:45, Bill Fischofer <bill.fischofer@linaro.org> wrote:

> OK, thanks.  That certainly matches the step-by-step semantics Petri
> proposed.  We'll have to discuss what, if anything, more we need regarding
> ORDERED queues post-v1.0 next year.
>
> Bill
>
> On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef <giladb@ezchip.com>
> wrote:
>
>>
>>
>>
>>
>> What NPS does in HW is that each buffer you dequeue actually stays on the
>> queue (but is marked as consumed by the application) and you are given a
>> "tag" for it.
>>
>>
>>
>> When I want to queue the buffer to the next, I give the HW the tag I got,
>> not the buffer handle and what the HW does is only than fully dequeue the
>> buffer from the original queue before queuing it into the new queue – but
>> only after all preceding (older) buffers on the original queue have been
>> fully dequeued,
>>
>>
>>
>> Free of a buffer causes a dequeue from the original queue as well.
>>
>>
>>
>> And I can also give release a buffer from the original queue (give the HW
>> the tag back and say "you don't have to wait for this buffer any longer).
>>
>>
>>
>> I hope this helps,
>>
>> Gilad
>>
>>
>>
>> *Gilad Ben-Yossef*
>>
>> Software Architect
>>
>> EZchip Technologies Ltd.
>>
>> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>>
>> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
>> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>>
>> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>>
>>
>>
>> *"Ethernet always wins."*
>>
>>         — Andy Bechtolsheim
>>
>>
>>
>> *From:* Bill Fischofer [mailto:bill.fischofer@linaro.org]
>> *Sent:* Sunday, November 02, 2014 5:44 PM
>> *To:* Gilad Ben Yossef
>> *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org
>>
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> Thanks Gilad.  Can you elaborate on that a bit more?  I can understand
>> how step-by-step would be potentially easier to implement, but does it also
>> capture the majority of expected application use cases?
>>
>>
>>
>> A good confirmation that this is the correct approach for v1.0 though.
>>
>>
>>
>> On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef <giladb@ezchip.com>
>> wrote:
>>
>>
>>
>> For what it's worth, as a SoC vendor rep. that has ordered queue in HW,
>> Petri's definition is actually preferred for us even going forward J
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Gilad
>>
>>
>>
>> *Gilad Ben-Yossef*
>>
>> Software Architect
>>
>> EZchip Technologies Ltd.
>>
>> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>>
>> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
>> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>>
>> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>>
>>
>>
>> *"Ethernet always wins."*
>>
>>         — Andy Bechtolsheim
>>
>>
>>
>> *From:* lng-odp-bounces@lists.linaro.org [mailto:
>> lng-odp-bounces@lists.linaro.org] *On Behalf Of *Bill Fischofer
>> *Sent:* Friday, October 31, 2014 2:59 PM
>> *To:* Savolainen, Petri (NSN - FI/Espoo)
>> *Cc:* lng-odp@lists.linaro.org
>>
>>
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> This may well be a reasonable restriction for ODP v1.0 but I believe it's
>> something we need to put on the list for "production grade" improvements
>> for 2015.
>>
>>
>>
>> Bill
>>
>>
>>
>> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
>> petri.savolainen@nsn.com> wrote:
>>
>> Yes, it’s step-by-step and I think it’s the level of ordering we need for
>> v1.0. Most SoCs can implement it, even when  the HW scheduler would not
>> have order support but only atomic/parallel. This way defined atomic
>> scheduling can be used to implement functionality correct ordered queues,
>> the throughput is not improved but it functions correctly.
>>
>>
>>
>> -Petri
>>
>>
>>
>>
>>
>> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
>> *Sent:* Friday, October 31, 2014 2:48 PM
>> *To:* Alexandru Badicioiu
>> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> I can well imagine the step-by-step order preservation to be simpler to
>> implement (in SW) but it would also seem to limit performance since the
>> only way to ensure end-to-end order preservation would be if each
>> intermediate queue from ingress to egress were an ordered queue.  If there
>> is a parallel queue anywhere in the chain that would break things.
>>
>>
>>
>> The question is: Is this restriction needed and/or sufficient for ODP
>> v1.0?
>>
>>
>>
>> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
>> alexandru.badicioiu@linaro.org> wrote:
>>
>> "+ * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call"
>>
>>
>>
>> Is this assuming that the order will be restored always at the next
>> enqueue? I think there should be an option to explicitly indicate if the
>> next enqueue is supposed to restore the order or not, especially when
>> packets move from queue to queue. Ordered queues are costly compared with
>> the ordinary ones.
>>
>>
>>
>> Alex
>>
>>
>>
>> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
>> wrote:
>>
>> Improved atomic and ordered synchronisation definitions. Added
>> order skip function prototype.
>>
>> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>>
>> ---
>> This is the ordered queue definition (in patch format) promised
>> in the call yesterday.
>> ---
>>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>>  platform/linux-generic/include/api/odp_schedule.h | 45
>> ++++++++++++++++++-----
>>  2 files changed, 64 insertions(+), 12 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/api/odp_queue.h
>> b/platform/linux-generic/include/api/odp_queue.h
>> index b8ac4bb..c0c3969 100644
>> --- a/platform/linux-generic/include/api/odp_queue.h
>> +++ b/platform/linux-generic/include/api/odp_queue.h
>> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>>  typedef int odp_schedule_sync_t;
>>
>>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
>> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
>> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
>> +
>> +/**
>> + * Atomic queue synchronisation
>> + *
>> + * The scheduler gives buffers from a queue to a single core at a time.
>> This
>> + * serialises processing of the buffers from the source queue and helps
>> user to
>> + * avoid SW locking. Another schedule call will implicitely release the
>> atomic
>> + * synchronisation of the source queue and free the scheduler to give
>> buffers
>> + * from the queue to other cores.
>> + *
>> + * User can hint the scheduler to release the atomic synchronisation
>> early with
>> + * odp_schedule_release_atomic().
>> + */
>> +#define ODP_SCHED_SYNC_ATOMIC   1
>> +
>> +/**
>> + * Ordered queue synchronisation
>> + *
>> + * The scheduler may give out buffers to multiple cores for parallel
>> processing.
>> + * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call.
>> Buffers
>> + * from the same ordered (source) queue appear in their original order
>> when
>> + * dequeued from a destination queue. The destination queue type
>> (POLL/SCHED) or
>> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
>> + *
>> + * User can command the scheduler to skip ordering of a buffer with
>> + * odp_schedule_skip_order().
>> + */
>> +#define ODP_SCHED_SYNC_ORDERED  2
>>
>>  /** Default queue synchronisation */
>>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
>> diff --git a/platform/linux-generic/include/api/odp_schedule.h
>> b/platform/linux-generic/include/api/odp_schedule.h
>> index 91fec10..2a1a642 100644
>> --- a/platform/linux-generic/include/api/odp_schedule.h
>> +++ b/platform/linux-generic/include/api/odp_schedule.h
>> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   * for a buffer according to the wait parameter setting. Returns
>>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>>
>>  /**
>>   * Schedule one buffer
>> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * User can exit the schedule loop without first calling
>> odp_schedule_pause().
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>>
>>
>>  /**
>> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * Like odp_schedule(), but returns multiple buffers from a queue.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * @return Number of buffers outputed (0 ... num)
>>   */
>> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
>> out_buf[],
>> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
>> out_buf[],
>>                        unsigned int num);
>>
>>  /**
>> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>>  void odp_schedule_resume(void);
>>
>>  /**
>> - * Release currently hold atomic context
>> + * Release the current atomic context
>> + *
>> + * This calls is valid only when the source queue has ATOMIC
>> synchronisation. It
>> + * hints the scheduler that the user has completed processing the
>> critical
>> + * section that needs atomic synchronisation. After the call, the
>> scheduler is
>> + * allowed to give next buffer from the same queue to another core.
>> + *
>> + * Usage of the call may increase parallelism and thus system
>> performance, but
>> + * carefulness is needed when splitting processing into critical vs.
>> + * non-critical sections.
>>   */
>>  void odp_schedule_release_atomic(void);
>>
>>  /**
>> + * Skip ordering of the buffer
>> + *
>> + * This calls is valid only when source queue has ORDERED
>> synchronisation. It
>> + * commands the scheduler to skip ordering of the buffer. Scheduler
>> maintains
>> + * ordering between two queues: source and destination. The dest
>> parameter
>> + * identifies the destination queue of that pair. After the call,
>> ordering is
>> + * not maintained for the buffer anymore but user still owns the buffer.
>> User
>> + * can e.g. store it, free it or enqueue it (to the same or another
>> queue).
>> + *
>> + * @param dest    Destination queue
>> + * @param buf     Buffer
>> + */
>> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
>> +
>> +
>> +/**
>>   * Number of scheduling priorities
>>   *
>>   * @return Number of scheduling priorities
>> --
>> 2.1.1
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>>
>>
>>
>>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>

The specific case I was talking about is when a packet is dequeued from an
ordered queue and it needs fragmentation as part of the processing before
the next enqueue which is supposed to restore the order - for example IPsec
red side fragmentation (fragmentation before encryption) or fragmentation
before transmission on an interface having a MTU smaller than the packet
size.
The fragments of the same packet are ordered with respect to each other but
they can be interleaved with other packets from the same queue or with
fragments of another packet.
As we have skip order function which essentially tells the order
restoration logic to give up on the current expected packet in sequence and
move to the next packet we also need a way to tell that a sequence of
buffers share the same ordering information so the order restoration logic
treats them as a whole rather as separate packets.

Alex




On 4 November 2014 17:15, Gilad Ben Yossef <giladb@ezchip.com> wrote:

>
>
> I don't think it can use the first fragment place in the order. It can use
> the last fragment to arrive place in the queue.
>
> Otherwise you have to stop forwarding until all fragments arrive to keep
> the order… no?
>
>
>
> *Gilad Ben-Yossef*
>
> Software Architect
>
> EZchip Technologies Ltd.
>
> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>
> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>
> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>
>
>
> *"Ethernet always wins."*
>
>         — Andy Bechtolsheim
>
>
>
> *From:* Alexandru Badicioiu [mailto:alexandru.badicioiu@linaro.org]
> *Sent:* Tuesday, November 04, 2014 4:13 PM
> *To:* Bill Fischofer
> *Cc:* Gilad Ben Yossef; Savolainen, Petri (NSN - FI/Espoo);
> lng-odp@lists.linaro.org
>
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> In addition to skip order function I think we need to handle the case when
> a packet gets fragmented (e.g. IP fragmentation). In this case the
> fragments should share the original sequence number and the order
> restoration logic should be aware when the last fragment occurs.
>
>
>
> Alex
>
>
>
> On 3 November 2014 13:45, Bill Fischofer <bill.fischofer@linaro.org>
> wrote:
>
> OK, thanks.  That certainly matches the step-by-step semantics Petri
> proposed.  We'll have to discuss what, if anything, more we need regarding
> ORDERED queues post-v1.0 next year.
>
>
>
> Bill
>
>
>
> On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef <giladb@ezchip.com>
> wrote:
>
>
>
>
>
> What NPS does in HW is that each buffer you dequeue actually stays on the
> queue (but is marked as consumed by the application) and you are given a
> "tag" for it.
>
>
>
> When I want to queue the buffer to the next, I give the HW the tag I got,
> not the buffer handle and what the HW does is only than fully dequeue the
> buffer from the original queue before queuing it into the new queue – but
> only after all preceding (older) buffers on the original queue have been
> fully dequeued,
>
>
>
> Free of a buffer causes a dequeue from the original queue as well.
>
>
>
> And I can also give release a buffer from the original queue (give the HW
> the tag back and say "you don't have to wait for this buffer any longer).
>
>
>
> I hope this helps,
>
> Gilad
>
>
>
> *Gilad Ben-Yossef*
>
> Software Architect
>
> EZchip Technologies Ltd.
>
> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>
> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>
> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>
>
>
> *"Ethernet always wins."*
>
>         — Andy Bechtolsheim
>
>
>
> *From:* Bill Fischofer [mailto:bill.fischofer@linaro.org]
> *Sent:* Sunday, November 02, 2014 5:44 PM
> *To:* Gilad Ben Yossef
> *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org
>
>
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> Thanks Gilad.  Can you elaborate on that a bit more?  I can understand how
> step-by-step would be potentially easier to implement, but does it also
> capture the majority of expected application use cases?
>
>
>
> A good confirmation that this is the correct approach for v1.0 though.
>
>
>
> On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef <giladb@ezchip.com>
> wrote:
>
>
>
> For what it's worth, as a SoC vendor rep. that has ordered queue in HW,
> Petri's definition is actually preferred for us even going forward J
>
>
>
>
>
> Thanks,
>
> Gilad
>
>
>
> *Gilad Ben-Yossef*
>
> Software Architect
>
> EZchip Technologies Ltd.
>
> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>
> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>
> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>
>
>
> *"Ethernet always wins."*
>
>         — Andy Bechtolsheim
>
>
>
> *From:* lng-odp-bounces@lists.linaro.org [mailto:
> lng-odp-bounces@lists.linaro.org] *On Behalf Of *Bill Fischofer
> *Sent:* Friday, October 31, 2014 2:59 PM
> *To:* Savolainen, Petri (NSN - FI/Espoo)
> *Cc:* lng-odp@lists.linaro.org
>
>
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> This may well be a reasonable restriction for ODP v1.0 but I believe it's
> something we need to put on the list for "production grade" improvements
> for 2015.
>
>
>
> Bill
>
>
>
> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
> petri.savolainen@nsn.com> wrote:
>
> Yes, it’s step-by-step and I think it’s the level of ordering we need for
> v1.0. Most SoCs can implement it, even when  the HW scheduler would not
> have order support but only atomic/parallel. This way defined atomic
> scheduling can be used to implement functionality correct ordered queues,
> the throughput is not improved but it functions correctly.
>
>
>
> -Petri
>
>
>
>
>
> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
> *Sent:* Friday, October 31, 2014 2:48 PM
> *To:* Alexandru Badicioiu
> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>
>
>
> I can well imagine the step-by-step order preservation to be simpler to
> implement (in SW) but it would also seem to limit performance since the
> only way to ensure end-to-end order preservation would be if each
> intermediate queue from ingress to egress were an ordered queue.  If there
> is a parallel queue anywhere in the chain that would break things.
>
>
>
> The question is: Is this restriction needed and/or sufficient for ODP
> v1.0?
>
>
>
> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
> alexandru.badicioiu@linaro.org> wrote:
>
> "+ * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call"
>
>
>
> Is this assuming that the order will be restored always at the next
> enqueue? I think there should be an option to explicitly indicate if the
> next enqueue is supposed to restore the order or not, especially when
> packets move from queue to queue. Ordered queues are costly compared with
> the ordinary ones.
>
>
>
> Alex
>
>
>
> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
> wrote:
>
> Improved atomic and ordered synchronisation definitions. Added
> order skip function prototype.
>
> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>
> ---
> This is the ordered queue definition (in patch format) promised
> in the call yesterday.
> ---
>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>  platform/linux-generic/include/api/odp_schedule.h | 45
> ++++++++++++++++++-----
>  2 files changed, 64 insertions(+), 12 deletions(-)
>
> diff --git a/platform/linux-generic/include/api/odp_queue.h
> b/platform/linux-generic/include/api/odp_queue.h
> index b8ac4bb..c0c3969 100644
> --- a/platform/linux-generic/include/api/odp_queue.h
> +++ b/platform/linux-generic/include/api/odp_queue.h
> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>  typedef int odp_schedule_sync_t;
>
>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
> +
> +/**
> + * Atomic queue synchronisation
> + *
> + * The scheduler gives buffers from a queue to a single core at a time.
> This
> + * serialises processing of the buffers from the source queue and helps
> user to
> + * avoid SW locking. Another schedule call will implicitely release the
> atomic
> + * synchronisation of the source queue and free the scheduler to give
> buffers
> + * from the queue to other cores.
> + *
> + * User can hint the scheduler to release the atomic synchronisation
> early with
> + * odp_schedule_release_atomic().
> + */
> +#define ODP_SCHED_SYNC_ATOMIC   1
> +
> +/**
> + * Ordered queue synchronisation
> + *
> + * The scheduler may give out buffers to multiple cores for parallel
> processing.
> + * The original enqueue order of the source queue is maintained when
> buffers are
> + * enqueued to their destination queue(s) before another schedule call.
> Buffers
> + * from the same ordered (source) queue appear in their original order
> when
> + * dequeued from a destination queue. The destination queue type
> (POLL/SCHED) or
> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
> + *
> + * User can command the scheduler to skip ordering of a buffer with
> + * odp_schedule_skip_order().
> + */
> +#define ODP_SCHED_SYNC_ORDERED  2
>
>  /** Default queue synchronisation */
>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
> diff --git a/platform/linux-generic/include/api/odp_schedule.h
> b/platform/linux-generic/include/api/odp_schedule.h
> index 91fec10..2a1a642 100644
> --- a/platform/linux-generic/include/api/odp_schedule.h
> +++ b/platform/linux-generic/include/api/odp_schedule.h
> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   * for a buffer according to the wait parameter setting. Returns
>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>
>  /**
>   * Schedule one buffer
> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * User can exit the schedule loop without first calling
> odp_schedule_pause().
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
> wait);
>   *
>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>   */
> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>
>
>  /**
> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * Like odp_schedule(), but returns multiple buffers from a queue.
>   *
> - * @param from    Output parameter for the source queue (where the buffer
> was
> - *                dequeued from). Ignored if NULL.
> + * @param src     The source queue (output). Indicates from which queue
> the
> + *                buffer was dequeued. Ignored if NULL.
>   * @param wait    Minimum time to wait for a buffer. Waits infinitely, if
> set to
>   *                ODP_SCHED_WAIT. Does not wait, if set to
> ODP_SCHED_NO_WAIT.
>   *                Use odp_schedule_wait_time() to convert time to other
> wait
> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
> uint64_t wait);
>   *
>   * @return Number of buffers outputed (0 ... num)
>   */
> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
> out_buf[],
> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
> out_buf[],
>                        unsigned int num);
>
>  /**
> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>  void odp_schedule_resume(void);
>
>  /**
> - * Release currently hold atomic context
> + * Release the current atomic context
> + *
> + * This calls is valid only when the source queue has ATOMIC
> synchronisation. It
> + * hints the scheduler that the user has completed processing the critical
> + * section that needs atomic synchronisation. After the call, the
> scheduler is
> + * allowed to give next buffer from the same queue to another core.
> + *
> + * Usage of the call may increase parallelism and thus system
> performance, but
> + * carefulness is needed when splitting processing into critical vs.
> + * non-critical sections.
>   */
>  void odp_schedule_release_atomic(void);
>
>  /**
> + * Skip ordering of the buffer
> + *
> + * This calls is valid only when source queue has ORDERED
> synchronisation. It
> + * commands the scheduler to skip ordering of the buffer. Scheduler
> maintains
> + * ordering between two queues: source and destination. The dest parameter
> + * identifies the destination queue of that pair. After the call,
> ordering is
> + * not maintained for the buffer anymore but user still owns the buffer.
> User
> + * can e.g. store it, free it or enqueue it (to the same or another
> queue).
> + *
> + * @param dest    Destination queue
> + * @param buf     Buffer
> + */
> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
> +
> +
> +/**
>   * Number of scheduling priorities
>   *
>   * @return Number of scheduling priorities
> --
> 2.1.1
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>

Alll we want to get this change in to help define the 1.0 API, can we get
reviews finalised for it.

On 5 November 2014 03:00, Alexandru Badicioiu <
alexandru.badicioiu@linaro.org> wrote:

> The specific case I was talking about is when a packet is dequeued from an
> ordered queue and it needs fragmentation as part of the processing before
> the next enqueue which is supposed to restore the order - for example IPsec
> red side fragmentation (fragmentation before encryption) or fragmentation
> before transmission on an interface having a MTU smaller than the packet
> size.
> The fragments of the same packet are ordered with respect to each other
> but they can be interleaved with other packets from the same queue or with
> fragments of another packet.
> As we have skip order function which essentially tells the order
> restoration logic to give up on the current expected packet in sequence and
> move to the next packet we also need a way to tell that a sequence of
> buffers share the same ordering information so the order restoration logic
> treats them as a whole rather as separate packets.
>
> Alex
>
>
>
>
> On 4 November 2014 17:15, Gilad Ben Yossef <giladb@ezchip.com> wrote:
>
>>
>>
>> I don't think it can use the first fragment place in the order. It can
>> use the last fragment to arrive place in the queue.
>>
>> Otherwise you have to stop forwarding until all fragments arrive to keep
>> the order… no?
>>
>>
>>
>> *Gilad Ben-Yossef*
>>
>> Software Architect
>>
>> EZchip Technologies Ltd.
>>
>> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>>
>> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
>> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>>
>> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>>
>>
>>
>> *"Ethernet always wins."*
>>
>>         — Andy Bechtolsheim
>>
>>
>>
>> *From:* Alexandru Badicioiu [mailto:alexandru.badicioiu@linaro.org]
>> *Sent:* Tuesday, November 04, 2014 4:13 PM
>> *To:* Bill Fischofer
>> *Cc:* Gilad Ben Yossef; Savolainen, Petri (NSN - FI/Espoo);
>> lng-odp@lists.linaro.org
>>
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> In addition to skip order function I think we need to handle the case
>> when a packet gets fragmented (e.g. IP fragmentation). In this case the
>> fragments should share the original sequence number and the order
>> restoration logic should be aware when the last fragment occurs.
>>
>>
>>
>> Alex
>>
>>
>>
>> On 3 November 2014 13:45, Bill Fischofer <bill.fischofer@linaro.org>
>> wrote:
>>
>> OK, thanks.  That certainly matches the step-by-step semantics Petri
>> proposed.  We'll have to discuss what, if anything, more we need regarding
>> ORDERED queues post-v1.0 next year.
>>
>>
>>
>> Bill
>>
>>
>>
>> On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef <giladb@ezchip.com>
>> wrote:
>>
>>
>>
>>
>>
>> What NPS does in HW is that each buffer you dequeue actually stays on the
>> queue (but is marked as consumed by the application) and you are given a
>> "tag" for it.
>>
>>
>>
>> When I want to queue the buffer to the next, I give the HW the tag I got,
>> not the buffer handle and what the HW does is only than fully dequeue the
>> buffer from the original queue before queuing it into the new queue – but
>> only after all preceding (older) buffers on the original queue have been
>> fully dequeued,
>>
>>
>>
>> Free of a buffer causes a dequeue from the original queue as well.
>>
>>
>>
>> And I can also give release a buffer from the original queue (give the HW
>> the tag back and say "you don't have to wait for this buffer any longer).
>>
>>
>>
>> I hope this helps,
>>
>> Gilad
>>
>>
>>
>> *Gilad Ben-Yossef*
>>
>> Software Architect
>>
>> EZchip Technologies Ltd.
>>
>> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>>
>> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
>> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>>
>> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>>
>>
>>
>> *"Ethernet always wins."*
>>
>>         — Andy Bechtolsheim
>>
>>
>>
>> *From:* Bill Fischofer [mailto:bill.fischofer@linaro.org]
>> *Sent:* Sunday, November 02, 2014 5:44 PM
>> *To:* Gilad Ben Yossef
>> *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org
>>
>>
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> Thanks Gilad.  Can you elaborate on that a bit more?  I can understand
>> how step-by-step would be potentially easier to implement, but does it also
>> capture the majority of expected application use cases?
>>
>>
>>
>> A good confirmation that this is the correct approach for v1.0 though.
>>
>>
>>
>> On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef <giladb@ezchip.com>
>> wrote:
>>
>>
>>
>> For what it's worth, as a SoC vendor rep. that has ordered queue in HW,
>> Petri's definition is actually preferred for us even going forward J
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Gilad
>>
>>
>>
>> *Gilad Ben-Yossef*
>>
>> Software Architect
>>
>> EZchip Technologies Ltd.
>>
>> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
>>
>> Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
>> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
>>
>> Email: giladb@ezchip.com, Web: http://www.ezchip.com
>>
>>
>>
>> *"Ethernet always wins."*
>>
>>         — Andy Bechtolsheim
>>
>>
>>
>> *From:* lng-odp-bounces@lists.linaro.org [mailto:
>> lng-odp-bounces@lists.linaro.org] *On Behalf Of *Bill Fischofer
>> *Sent:* Friday, October 31, 2014 2:59 PM
>> *To:* Savolainen, Petri (NSN - FI/Espoo)
>> *Cc:* lng-odp@lists.linaro.org
>>
>>
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> This may well be a reasonable restriction for ODP v1.0 but I believe it's
>> something we need to put on the list for "production grade" improvements
>> for 2015.
>>
>>
>>
>> Bill
>>
>>
>>
>> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) <
>> petri.savolainen@nsn.com> wrote:
>>
>> Yes, it’s step-by-step and I think it’s the level of ordering we need for
>> v1.0. Most SoCs can implement it, even when  the HW scheduler would not
>> have order support but only atomic/parallel. This way defined atomic
>> scheduling can be used to implement functionality correct ordered queues,
>> the throughput is not improved but it functions correctly.
>>
>>
>>
>> -Petri
>>
>>
>>
>>
>>
>> *From:* ext Bill Fischofer [mailto:bill.fischofer@linaro.org]
>> *Sent:* Friday, October 31, 2014 2:48 PM
>> *To:* Alexandru Badicioiu
>> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org
>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
>>
>>
>>
>> I can well imagine the step-by-step order preservation to be simpler to
>> implement (in SW) but it would also seem to limit performance since the
>> only way to ensure end-to-end order preservation would be if each
>> intermediate queue from ingress to egress were an ordered queue.  If there
>> is a parallel queue anywhere in the chain that would break things.
>>
>>
>>
>> The question is: Is this restriction needed and/or sufficient for ODP
>> v1.0?
>>
>>
>>
>> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu <
>> alexandru.badicioiu@linaro.org> wrote:
>>
>> "+ * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call"
>>
>>
>>
>> Is this assuming that the order will be restored always at the next
>> enqueue? I think there should be an option to explicitly indicate if the
>> next enqueue is supposed to restore the order or not, especially when
>> packets move from queue to queue. Ordered queues are costly compared with
>> the ordinary ones.
>>
>>
>>
>> Alex
>>
>>
>>
>> On 31 October 2014 14:25, Petri Savolainen <petri.savolainen@linaro.org>
>> wrote:
>>
>> Improved atomic and ordered synchronisation definitions. Added
>> order skip function prototype.
>>
>> Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org>
>>
>> ---
>> This is the ordered queue definition (in patch format) promised
>> in the call yesterday.
>> ---
>>  platform/linux-generic/include/api/odp_queue.h    | 31 +++++++++++++++-
>>  platform/linux-generic/include/api/odp_schedule.h | 45
>> ++++++++++++++++++-----
>>  2 files changed, 64 insertions(+), 12 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/api/odp_queue.h
>> b/platform/linux-generic/include/api/odp_queue.h
>> index b8ac4bb..c0c3969 100644
>> --- a/platform/linux-generic/include/api/odp_queue.h
>> +++ b/platform/linux-generic/include/api/odp_queue.h
>> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t;
>>  typedef int odp_schedule_sync_t;
>>
>>  #define ODP_SCHED_SYNC_NONE     0  /**< Queue not synchronised */
>> -#define ODP_SCHED_SYNC_ATOMIC   1  /**< Atomic queue */
>> -#define ODP_SCHED_SYNC_ORDERED  2  /**< Ordered queue */
>> +
>> +/**
>> + * Atomic queue synchronisation
>> + *
>> + * The scheduler gives buffers from a queue to a single core at a time.
>> This
>> + * serialises processing of the buffers from the source queue and helps
>> user to
>> + * avoid SW locking. Another schedule call will implicitely release the
>> atomic
>> + * synchronisation of the source queue and free the scheduler to give
>> buffers
>> + * from the queue to other cores.
>> + *
>> + * User can hint the scheduler to release the atomic synchronisation
>> early with
>> + * odp_schedule_release_atomic().
>> + */
>> +#define ODP_SCHED_SYNC_ATOMIC   1
>> +
>> +/**
>> + * Ordered queue synchronisation
>> + *
>> + * The scheduler may give out buffers to multiple cores for parallel
>> processing.
>> + * The original enqueue order of the source queue is maintained when
>> buffers are
>> + * enqueued to their destination queue(s) before another schedule call.
>> Buffers
>> + * from the same ordered (source) queue appear in their original order
>> when
>> + * dequeued from a destination queue. The destination queue type
>> (POLL/SCHED) or
>> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited.
>> + *
>> + * User can command the scheduler to skip ordering of a buffer with
>> + * odp_schedule_skip_order().
>> + */
>> +#define ODP_SCHED_SYNC_ORDERED  2
>>
>>  /** Default queue synchronisation */
>>  #define ODP_SCHED_SYNC_DEFAULT  ODP_SCHED_SYNC_ATOMIC
>> diff --git a/platform/linux-generic/include/api/odp_schedule.h
>> b/platform/linux-generic/include/api/odp_schedule.h
>> index 91fec10..2a1a642 100644
>> --- a/platform/linux-generic/include/api/odp_schedule.h
>> +++ b/platform/linux-generic/include/api/odp_schedule.h
>> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   * for a buffer according to the wait parameter setting. Returns
>>   * ODP_BUFFER_INVALID if reaches end of the wait period.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait);
>>
>>  /**
>>   * Schedule one buffer
>> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * User can exit the schedule loop without first calling
>> odp_schedule_pause().
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t
>> wait);
>>   *
>>   * @return Next highest priority buffer, or ODP_BUFFER_INVALID
>>   */
>> -odp_buffer_t odp_schedule_one(odp_queue_t *from, uint64_t wait);
>> +odp_buffer_t odp_schedule_one(odp_queue_t *src, uint64_t wait);
>>
>>
>>  /**
>> @@ -93,8 +93,8 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * Like odp_schedule(), but returns multiple buffers from a queue.
>>   *
>> - * @param from    Output parameter for the source queue (where the
>> buffer was
>> - *                dequeued from). Ignored if NULL.
>> + * @param src     The source queue (output). Indicates from which queue
>> the
>> + *                buffer was dequeued. Ignored if NULL.
>>   * @param wait    Minimum time to wait for a buffer. Waits infinitely,
>> if set to
>>   *                ODP_SCHED_WAIT. Does not wait, if set to
>> ODP_SCHED_NO_WAIT.
>>   *                Use odp_schedule_wait_time() to convert time to other
>> wait
>> @@ -104,7 +104,7 @@ odp_buffer_t odp_schedule_one(odp_queue_t *from,
>> uint64_t wait);
>>   *
>>   * @return Number of buffers outputed (0 ... num)
>>   */
>> -int odp_schedule_multi(odp_queue_t *from, uint64_t wait, odp_buffer_t
>> out_buf[],
>> +int odp_schedule_multi(odp_queue_t *src, uint64_t wait, odp_buffer_t
>> out_buf[],
>>                        unsigned int num);
>>
>>  /**
>> @@ -129,11 +129,36 @@ void odp_schedule_pause(void);
>>  void odp_schedule_resume(void);
>>
>>  /**
>> - * Release currently hold atomic context
>> + * Release the current atomic context
>> + *
>> + * This calls is valid only when the source queue has ATOMIC
>> synchronisation. It
>> + * hints the scheduler that the user has completed processing the
>> critical
>> + * section that needs atomic synchronisation. After the call, the
>> scheduler is
>> + * allowed to give next buffer from the same queue to another core.
>> + *
>> + * Usage of the call may increase parallelism and thus system
>> performance, but
>> + * carefulness is needed when splitting processing into critical vs.
>> + * non-critical sections.
>>   */
>>  void odp_schedule_release_atomic(void);
>>
>>  /**
>> + * Skip ordering of the buffer
>> + *
>> + * This calls is valid only when source queue has ORDERED
>> synchronisation. It
>> + * commands the scheduler to skip ordering of the buffer. Scheduler
>> maintains
>> + * ordering between two queues: source and destination. The dest
>> parameter
>> + * identifies the destination queue of that pair. After the call,
>> ordering is
>> + * not maintained for the buffer anymore but user still owns the buffer.
>> User
>> + * can e.g. store it, free it or enqueue it (to the same or another
>> queue).
>> + *
>> + * @param dest    Destination queue
>> + * @param buf     Buffer
>> + */
>> +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf);
>> +
>> +
>> +/**
>>   * Number of scheduling priorities
>>   *
>>   * @return Number of scheduling priorities
>> --
>> 2.1.1
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> http://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/lng-odp
>
>