From patchwork Fri Aug 24 14:00:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 145069 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1304502ljw; Fri, 24 Aug 2018 07:02:07 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY9faB2mhnMDyAYCc2T+ypVzUihskm+rGIopsqswgoSpCACFtUGQpVfo2WAemiC0Bm1i6+0 X-Received: by 2002:a5b:950:: with SMTP id x16-v6mr1075329ybq.67.1535119327769; Fri, 24 Aug 2018 07:02:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535119327; cv=none; d=google.com; s=arc-20160816; b=R+loz9QKCTx7GrXagBeFlnCCb87ynK4EZMu6ZEFOeILIKvepjWpyWw0Az2Ei9yEGAM Q7mOBxdohZIPMNMMCIB/FtfiySFBxuZ+ucGbE08+226paRb6OIWsCJnr3ebqgA2LSrH1 s+cI5i22Hfiwuqxa9sXaDfZBDbnXD6YFvoljJQ8t6PqnlkKYQdoDpANqlXQUNq8huALW 6x8VCUpkOvqSfPj0jQey6JKpZnpIbNxzcOyuPuGTfPbxhTjzUO7waB9F1szqJ/Y73q/w UfYpyiVIXuaftNhmU5zBlakPDE/oFzGK31hFR6QIjfWPnXTcb36L2yOUt/MvgBtbMZ7M PrbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=Knh+Y+48PVCmpENlD3T14fXDZ0hKKYc31ITJTFV8rEc=; b=mdeeQMcnWXUL18Ddnu1ZPf+cgGEdyt2bbdqaZcwwXLJkyNWfuzhaoglCVMIBpgkg9t 3w9JgZpV+NqrskKUY4pyE8Zn/yV1nkbSXv8run1AIIsiDMA/sFaGaOJnx/uKEHX3gXae UFpEIkCacgO7gm3YBWHYCCGZ4efmG5e8iO23saQgRjvwHCcVBp3evSp7kXsxCBqzoFDD x16iqUPOWJ3K4FtlbIx37TLtMKjUqC3aM/xXGvG2HbT4Gf5x7yolS+mUWt+ZjBeAulrh NV7q0vtnUYEL/2BuOLp52awrmYPqnoGrI1Hinmdgrr6lBVPxt+/OKF/C4T49XdCvUHsY UL8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id u41-v6si1592152qvc.146.2018.08.24.07.02.07; Fri, 24 Aug 2018 07:02:07 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 5AFDD60F00; Fri, 24 Aug 2018 14:02:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 0CA7860694; Fri, 24 Aug 2018 14:00:31 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id AD272607C8; Fri, 24 Aug 2018 14:00:16 +0000 (UTC) Received: from forward103p.mail.yandex.net (forward103p.mail.yandex.net [77.88.28.106]) by lists.linaro.org (Postfix) with ESMTPS id C9DA1607EC for ; Fri, 24 Aug 2018 14:00:11 +0000 (UTC) Received: from mxback3j.mail.yandex.net (mxback3j.mail.yandex.net [IPv6:2a02:6b8:0:1619::10c]) by forward103p.mail.yandex.net (Yandex) with ESMTP id 58E682185771 for ; Fri, 24 Aug 2018 17:00:09 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback3j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 1jMvusfWvl-09QmeGUe; Fri, 24 Aug 2018 17:00:09 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 2CVvDjLgK3-08jecAE5; Fri, 24 Aug 2018 17:00:08 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 24 Aug 2018 14:00:03 +0000 Message-Id: <1535119206-23556-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> References: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 683 Subject: [lng-odp] [PATCH v1 1/4] linux-gen: ring_mpmc: new multi-producer, multi-consumer ring X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen The ring is similar to ring_internal.h, but checks for ring fullness. This ring can be used for storing events in a queue as enqueues can be tried on an already full queue. Signed-off-by: Petri Savolainen --- /** Email created from pull request 683 (psavol:master-queue-lockless-enqdeq-3) ** https://github.com/Linaro/odp/pull/683 ** Patch: https://github.com/Linaro/odp/pull/683.patch ** Base sha: 989df5d2f97ab4711328b11282dcc743f5740e00 ** Merge commit sha: 28073c54671148efdd01c9cf38c1a235d5a133f0 **/ platform/linux-generic/Makefile.am | 1 + .../include/odp_ring_mpmc_internal.h | 169 ++++++++++++++++++ 2 files changed, 170 insertions(+) create mode 100644 platform/linux-generic/include/odp_ring_mpmc_internal.h diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 80f968756..ab0b755d5 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -127,6 +127,7 @@ noinst_HEADERS = \ include/odp_queue_lf.h \ include/odp_queue_scalable_internal.h \ include/odp_ring_internal.h \ + include/odp_ring_mpmc_internal.h \ include/odp_ring_spsc_internal.h \ include/odp_ring_st_internal.h \ include/odp_schedule_if.h \ diff --git a/platform/linux-generic/include/odp_ring_mpmc_internal.h b/platform/linux-generic/include/odp_ring_mpmc_internal.h new file mode 100644 index 000000000..74bbb8fc7 --- /dev/null +++ b/platform/linux-generic/include/odp_ring_mpmc_internal.h @@ -0,0 +1,169 @@ +/* Copyright (c) 2018, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_RING_MPMC_INTERNAL_H_ +#define ODP_RING_MPMC_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include +#include +#include +#include +#include +#include + +/* Ring of uint32_t data + * + * Ring stores head and tail counters. Ring indexes are formed from these + * counters with a mask (mask = ring_size - 1), which requires that ring size + * must be a power of two. + * + * 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 + * +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+ + * | E | E | | | | | | | | | | E | E | E | E | E | + * +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+ + * ^ ^ ^ ^ + * | | | | + * r_tail r_head w_tail w_head + * + */ +typedef struct { + odp_atomic_u32_t ODP_ALIGNED_CACHE r_head; + odp_atomic_u32_t r_tail; + + odp_atomic_u32_t ODP_ALIGNED_CACHE w_head; + odp_atomic_u32_t w_tail; + +} ring_mpmc_t; + +static inline int ring_mpmc_cas_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, uint32_t new_val) +{ + return __atomic_compare_exchange_n(&atom->v, old_val, new_val, + 0 /* strong */, + __ATOMIC_RELAXED, + __ATOMIC_RELAXED); +} + +/* Initialize ring */ +static inline void ring_mpmc_init(ring_mpmc_t *ring) +{ + odp_atomic_init_u32(&ring->w_head, 0); + odp_atomic_init_u32(&ring->w_tail, 0); + odp_atomic_init_u32(&ring->r_head, 0); + odp_atomic_init_u32(&ring->r_tail, 0); +} + +/* Dequeue data from the ring head. Num is smaller than ring size. */ +static inline uint32_t ring_mpmc_deq_multi(ring_mpmc_t *ring, + uint32_t *ring_data, + uint32_t ring_mask, + uint32_t data[], + uint32_t num) +{ + uint32_t old_head, new_head, w_tail, num_data, i; + + /* Load acquires ensure that w_tail load happens after r_head load, + * and thus r_head value is always behind or equal to w_tail value. + * When CAS operation succeeds, this thread owns data between old + * and new r_head. */ + do { + old_head = odp_atomic_load_acq_u32(&ring->r_head); + odp_prefetch(&ring_data[(old_head + 1) & ring_mask]); + w_tail = odp_atomic_load_acq_u32(&ring->w_tail); + num_data = w_tail - old_head; + + /* Ring is empty */ + if (num_data == 0) + return 0; + + /* Try to take all available */ + if (num > num_data) + num = num_data; + + new_head = old_head + num; + + } while (odp_unlikely(ring_mpmc_cas_u32(&ring->r_head, &old_head, + new_head) == 0)); + + /* Read data. This will not move above load acquire of r_head. */ + for (i = 0; i < num; i++) + data[i] = ring_data[(old_head + 1 + i) & ring_mask]; + + /* Wait until other readers have updated the tail */ + while (odp_unlikely(odp_atomic_load_u32(&ring->r_tail) != old_head)) + odp_cpu_pause(); + + /* Release the new reader tail, writers acquire it. */ + odp_atomic_store_rel_u32(&ring->r_tail, new_head); + + return num; +} + +/* Enqueue multiple data into the ring tail. Num is smaller than ring size. */ +static inline uint32_t ring_mpmc_enq_multi(ring_mpmc_t *ring, + uint32_t *ring_data, + uint32_t ring_mask, + const uint32_t data[], + uint32_t num) +{ + uint32_t old_head, new_head, r_tail, num_free, i; + uint32_t size = ring_mask + 1; + + /* Load acquires ensure that w_head load happens after r_tail load, + * and thus r_tail value is always behind or equal to w_head value. + * When CAS operation succeeds, this thread owns data between old + * and new w_head. */ + do { + r_tail = odp_atomic_load_acq_u32(&ring->r_tail); + old_head = odp_atomic_load_acq_u32(&ring->w_head); + + num_free = size - (old_head - r_tail); + + /* Ring is full */ + if (num_free == 0) + return 0; + + /* Try to use all available */ + if (num > num_free) + num = num_free; + + new_head = old_head + num; + + } while (odp_unlikely(ring_mpmc_cas_u32(&ring->w_head, &old_head, + new_head) == 0)); + + /* Write data. This will not move above load acquire of w_head. */ + for (i = 0; i < num; i++) + ring_data[(old_head + 1 + i) & ring_mask] = data[i]; + + /* Wait until other writers have updated the tail */ + while (odp_unlikely(odp_atomic_load_u32(&ring->w_tail) != old_head)) + odp_cpu_pause(); + + /* Release the new writer tail, readers acquire it. */ + odp_atomic_store_rel_u32(&ring->w_tail, new_head); + + return num; +} + +/* Check if ring is empty */ +static inline int ring_mpmc_is_empty(ring_mpmc_t *ring) +{ + uint32_t head = odp_atomic_load_u32(&ring->r_head); + uint32_t tail = odp_atomic_load_u32(&ring->w_tail); + + return head == tail; +} + +#ifdef __cplusplus +} +#endif + +#endif From patchwork Fri Aug 24 14:00:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 145067 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1302728ljw; Fri, 24 Aug 2018 07:00:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZH0/pXKbnw5LNR+XjqN54BRCqnEpG6Xr1YGG69Rksd7Mmtnmrlwc3lDVspg8Urj0Yicmw1 X-Received: by 2002:ac8:2463:: with SMTP id d32-v6mr1832114qtd.41.1535119252822; Fri, 24 Aug 2018 07:00:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535119252; cv=none; d=google.com; s=arc-20160816; b=B0ez0fmFgpmEwSriGRMmqH30bP8w6f8/zNbIwVdkt0zME69u/9qGT79jlVXQnvNrhZ B/v3GJmtChzRXckGIVTt3g3hrypZ7EPpCaL92xF5Ka5RbjG9DJFgj5Odsy//HHp/T19I 3lljMsqC7wc1xeUcvzeLjb4bgjbv77jQWIyrAZ3GJCn+M7tDah7EAtclm+EbEIVuvd50 6oc0gwjFCDiTxAF9KJtx73FwMD7oD8R7BgBB6Ia0pqfZ5lMs9oT3b5yBjtzT/DwcWWkB xPPCH4w9zDKUQi/qOSot0leRRGaRdCiOsqSe47xOBGrcZ6kTp39CBXMF2IAJ6wdQcClG wN4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=VJSHsIQMSlq1OCS6u5z+CUzH3Dmtj0OboN9p132fa6Q=; b=S6SCerBsKSzHCHLKRc/BYNJdnNXiIqqsBdM0HyXkY7gR3QYi7zqzXc59iowfb3AuK7 B5pWjZziyYGAm5H6vFtV63YYBtXe3jYV9mgle0nyyk0OV0fz6v4Pjgp/UXUFU/otaSab y+SQkbdREtP8jowh9+WY/+Xs1cKQvtQli7jcMiMzdQakD+qlyQqx8MY1I4/0ZDWLe8Nx MzoIVntU9Y1rhODValvJ1PuY9NU2u/Yr1Rf8xfGoHH313HgolLYTdrXDdXnqtfCpMYmj soz4XRjzlnHZ+G2samUbIfsPPhA43LcUclj7L8kNACtlM+TPqzmvpIsjI9k4pI8AQOCG JPRw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id p1-v6si150390qta.346.2018.08.24.07.00.52; Fri, 24 Aug 2018 07:00:52 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 68D92608B7; Fri, 24 Aug 2018 14:00:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 5D7E260800; Fri, 24 Aug 2018 14:00:20 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 75E3F608B7; Fri, 24 Aug 2018 14:00:13 +0000 (UTC) Received: from forward102o.mail.yandex.net (forward102o.mail.yandex.net [37.140.190.182]) by lists.linaro.org (Postfix) with ESMTPS id 80332607C8 for ; Fri, 24 Aug 2018 14:00:11 +0000 (UTC) Received: from mxback6j.mail.yandex.net (mxback6j.mail.yandex.net [IPv6:2a02:6b8:0:1619::10f]) by forward102o.mail.yandex.net (Yandex) with ESMTP id 1BF045A04177 for ; Fri, 24 Aug 2018 17:00:10 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback6j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id ZM1x0AU2Q7-0ArCQCZw; Fri, 24 Aug 2018 17:00:10 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 2CVvDjLgK3-09jeHamU; Fri, 24 Aug 2018 17:00:09 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 24 Aug 2018 14:00:04 +0000 Message-Id: <1535119206-23556-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> References: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 683 Subject: [lng-odp] [PATCH v1 2/4] linux-gen: queue: use mpmc ring in plain queues X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Change plain queue implementation to use ring_mpmc instead of ticket lock and ring_st ring. Performance and scalability improves especially on 64 bit ARM. Signed-off-by: Petri Savolainen --- /** Email created from pull request 683 (psavol:master-queue-lockless-enqdeq-3) ** https://github.com/Linaro/odp/pull/683 ** Patch: https://github.com/Linaro/odp/pull/683.patch ** Base sha: 989df5d2f97ab4711328b11282dcc743f5740e00 ** Merge commit sha: 28073c54671148efdd01c9cf38c1a235d5a133f0 **/ .../include/odp_queue_basic_internal.h | 30 +++++++----- platform/linux-generic/odp_queue_basic.c | 48 +++++++------------ 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/platform/linux-generic/include/odp_queue_basic_internal.h b/platform/linux-generic/include/odp_queue_basic_internal.h index 15e49772c..46b747955 100644 --- a/platform/linux-generic/include/odp_queue_basic_internal.h +++ b/platform/linux-generic/include/odp_queue_basic_internal.h @@ -22,6 +22,7 @@ extern "C" { #include #include #include +#include #include #include #include @@ -33,22 +34,29 @@ extern "C" { #define QUEUE_STATUS_SCHED 4 struct queue_entry_s { - odp_ticketlock_t ODP_ALIGNED_CACHE lock; - union { - ring_st_t ring_st; - ring_spsc_t ring_spsc; - }; - int status; - + /* The first cache line is read only */ queue_enq_fn_t ODP_ALIGNED_CACHE enqueue; queue_deq_fn_t dequeue; queue_enq_multi_fn_t enqueue_multi; queue_deq_multi_fn_t dequeue_multi; - queue_deq_multi_fn_t orig_dequeue_multi; + uint32_t *ring_data; + uint32_t ring_mask; + uint32_t index; + odp_queue_t handle; + odp_queue_type_t type; + + /* MPMC ring (2 cache lines). */ + ring_mpmc_t ring_mpmc; - uint32_t index; - odp_queue_t handle; - odp_queue_type_t type; + odp_ticketlock_t lock; + union { + ring_st_t ring_st; + ring_spsc_t ring_spsc; + }; + + int status; + + queue_deq_multi_fn_t orig_dequeue_multi; odp_queue_param_t param; odp_pktin_queue_t pktin; odp_pktout_queue_t pktout; diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index e5d915643..8b9a70bbc 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -400,8 +400,10 @@ static int queue_destroy(odp_queue_t handle) if (queue->s.spsc) empty = ring_spsc_is_empty(&queue->s.ring_spsc); - else + else if (queue->s.type == ODP_QUEUE_TYPE_SCHED) empty = ring_st_is_empty(&queue->s.ring_st); + else + empty = ring_mpmc_is_empty(&queue->s.ring_mpmc); if (!empty) { UNLOCK(queue); @@ -490,28 +492,19 @@ static inline int _plain_queue_enq_multi(odp_queue_t handle, { queue_entry_t *queue; int ret, num_enq; - ring_st_t *ring_st; + ring_mpmc_t *ring_mpmc; uint32_t buf_idx[num]; queue = qentry_from_handle(handle); - ring_st = &queue->s.ring_st; + ring_mpmc = &queue->s.ring_mpmc; if (sched_fn->ord_enq_multi(handle, (void **)buf_hdr, num, &ret)) return ret; buffer_index_from_buf(buf_idx, buf_hdr, num); - LOCK(queue); - - if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { - UNLOCK(queue); - ODP_ERR("Bad queue status\n"); - return -1; - } - - num_enq = ring_st_enq_multi(ring_st, buf_idx, num); - - UNLOCK(queue); + num_enq = ring_mpmc_enq_multi(ring_mpmc, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); return num_enq; } @@ -521,23 +514,14 @@ static inline int _plain_queue_deq_multi(odp_queue_t handle, { int num_deq; queue_entry_t *queue; - ring_st_t *ring_st; + ring_mpmc_t *ring_mpmc; uint32_t buf_idx[num]; queue = qentry_from_handle(handle); - ring_st = &queue->s.ring_st; + ring_mpmc = &queue->s.ring_mpmc; - LOCK(queue); - - if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { - /* Bad queue, or queue has been destroyed. */ - UNLOCK(queue); - return -1; - } - - num_deq = ring_st_deq_multi(ring_st, buf_idx, num); - - UNLOCK(queue); + num_deq = ring_mpmc_deq_multi(ring_mpmc, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); if (num_deq == 0) return 0; @@ -883,13 +867,17 @@ static int queue_init(queue_entry_t *queue, const char *name, queue->s.dequeue = plain_queue_deq; queue->s.dequeue_multi = plain_queue_deq_multi; queue->s.orig_dequeue_multi = plain_queue_deq_multi; + + queue->s.ring_data = &queue_glb->ring_data[offset]; + queue->s.ring_mask = queue_size - 1; + ring_mpmc_init(&queue->s.ring_mpmc); + } else { queue->s.enqueue = sched_queue_enq; queue->s.enqueue_multi = sched_queue_enq_multi; + ring_st_init(&queue->s.ring_st, + &queue_glb->ring_data[offset], queue_size); } - - ring_st_init(&queue->s.ring_st, &queue_glb->ring_data[offset], - queue_size); } return 0; From patchwork Fri Aug 24 14:00:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 145068 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1303603ljw; Fri, 24 Aug 2018 07:01:30 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYwEWJyD5rsYQCcxuaMJgDIUur3l78Zxf1mZrBTQXtT/G8biWMuuUYl7n/ydeLZf8VyO5PV X-Received: by 2002:ac8:611c:: with SMTP id a28-v6mr1857735qtm.130.1535119290801; Fri, 24 Aug 2018 07:01:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535119290; cv=none; d=google.com; s=arc-20160816; b=o2TOUw0cnDFnyQfs3FWLsH4JdNehtnnz+0AL2KQjQm4+OCnq8yDcaKYq3P2wX9O5ag C22jNNijfrUmI6Lxj0z4HC536J1LISRZemFUr1tIwgQSkqONSF3HL6STHLCE30qQGLlL 5gc65O3Nhu2+TYS4pd13Bi2if9YAocEnCWx1lqnqbcmnoydfQbmVYXSSb3kc9Kk6e38s Xl4I5Z8P3Siyo1hv1GXfAQ+qeC+nNi6ObpJPQ9dnLyruMr7/yAYAAmxsngOeilEXWpow VoJAyW3lMgDmLeMXS38KVCv/KG588RgP+IaeQys1bVRDB8d0SEk6IjIFD79hmL0qRWU4 LXTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=mGilxw4b6WOpj2LyrNCgX0Hldl7PC8VY5ENdSElOq+g=; b=B32DjHS/2SjX04YHzetg9TPnnPcZ7HX1KuqpRj0nBpDGK8d8OZ+BBfMEayaMRZ2+ZH MYbA50rUgri3C1Zc8C9FrShiESSqIhLymxu4ZjdXRfVXEroapmPSRqF5PHdwjrNXz9BA AqLCXG8ytLpZjrsEJqooLHQOiqVECLsTOFgDcSNxAjR7k55WAvPfWFACbUwCik5J5r2t O/BeJ6u078cDrvL4dDL4I/881u7QenqCJQcRi+I5vpySO4DMH6JluUInLOw8dbTbXX0w acs4iB6IxqxJqb/8lvQ/p0lwVjUufaYWZ8usvXmD1pYfVW6lC9Gn5XIuU2x648OMyj7M 7YAg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id r7-v6si2024666qvm.102.2018.08.24.07.01.30; Fri, 24 Aug 2018 07:01:30 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 6056660F00; Fri, 24 Aug 2018 14:01:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id AF76260833; Fri, 24 Aug 2018 14:00:25 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 7C42D607C8; Fri, 24 Aug 2018 14:00:16 +0000 (UTC) Received: from forward100p.mail.yandex.net (forward100p.mail.yandex.net [77.88.28.100]) by lists.linaro.org (Postfix) with ESMTPS id 1131C60800 for ; Fri, 24 Aug 2018 14:00:12 +0000 (UTC) Received: from mxback15g.mail.yandex.net (mxback15g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:94]) by forward100p.mail.yandex.net (Yandex) with ESMTP id C62D6510519D for ; Fri, 24 Aug 2018 17:00:10 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback15g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id pGIWWdcLVi-0AG4V3vX; Fri, 24 Aug 2018 17:00:10 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 2CVvDjLgK3-0AjGME75; Fri, 24 Aug 2018 17:00:10 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 24 Aug 2018 14:00:05 +0000 Message-Id: <1535119206-23556-4-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> References: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 683 Subject: [lng-odp] [PATCH v1 3/4] linux-gen: ring_st: move ring mask and data pointer X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Store mask and data pointer in queue entry instead of ring structure. Data is constant and can be stored among other frequently used read only data. Also other ring type use the same variables. Signed-off-by: Petri Savolainen --- /** Email created from pull request 683 (psavol:master-queue-lockless-enqdeq-3) ** https://github.com/Linaro/odp/pull/683 ** Patch: https://github.com/Linaro/odp/pull/683.patch ** Base sha: 989df5d2f97ab4711328b11282dcc743f5740e00 ** Merge commit sha: 28073c54671148efdd01c9cf38c1a235d5a133f0 **/ .../include/odp_ring_st_internal.h | 34 ++++++++----------- platform/linux-generic/odp_queue_basic.c | 12 ++++--- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/platform/linux-generic/include/odp_ring_st_internal.h b/platform/linux-generic/include/odp_ring_st_internal.h index 5fb37d4ef..1bc18cda0 100644 --- a/platform/linux-generic/include/odp_ring_st_internal.h +++ b/platform/linux-generic/include/odp_ring_st_internal.h @@ -19,30 +19,25 @@ extern "C" { typedef struct { uint32_t head; uint32_t tail; - uint32_t mask; - uint32_t *data; - } ring_st_t; /* Initialize ring. Ring size must be a power of two. */ -static inline void ring_st_init(ring_st_t *ring, uint32_t *data, uint32_t size) +static inline void ring_st_init(ring_st_t *ring) { ring->head = 0; ring->tail = 0; - ring->mask = size - 1; - ring->data = data; } /* Dequeue data from the ring head. Max_num is smaller than ring size.*/ -static inline uint32_t ring_st_deq_multi(ring_st_t *ring, uint32_t data[], +static inline uint32_t ring_st_deq_multi(ring_st_t *ring, uint32_t *ring_data, + uint32_t ring_mask, uint32_t data[], uint32_t max_num) { - uint32_t head, tail, mask, idx; + uint32_t head, tail, idx; uint32_t num, i; head = ring->head; tail = ring->tail; - mask = ring->mask; num = tail - head; /* Empty */ @@ -52,11 +47,11 @@ static inline uint32_t ring_st_deq_multi(ring_st_t *ring, uint32_t data[], if (num > max_num) num = max_num; - idx = head & mask; + idx = head & ring_mask; for (i = 0; i < num; i++) { - data[i] = ring->data[idx]; - idx = (idx + 1) & mask; + data[i] = ring_data[idx]; + idx = (idx + 1) & ring_mask; } ring->head = head + num; @@ -65,16 +60,17 @@ static inline uint32_t ring_st_deq_multi(ring_st_t *ring, uint32_t data[], } /* Enqueue data into the ring tail. Num_data is smaller than ring size. */ -static inline uint32_t ring_st_enq_multi(ring_st_t *ring, const uint32_t data[], +static inline uint32_t ring_st_enq_multi(ring_st_t *ring, uint32_t *ring_data, + uint32_t ring_mask, + const uint32_t data[], uint32_t num_data) { - uint32_t head, tail, mask, size, idx; + uint32_t head, tail, size, idx; uint32_t num, i; head = ring->head; tail = ring->tail; - mask = ring->mask; - size = mask + 1; + size = ring_mask + 1; num = size - (tail - head); /* Full */ @@ -84,11 +80,11 @@ static inline uint32_t ring_st_enq_multi(ring_st_t *ring, const uint32_t data[], if (num > num_data) num = num_data; - idx = tail & mask; + idx = tail & ring_mask; for (i = 0; i < num; i++) { - ring->data[idx] = data[i]; - idx = (idx + 1) & mask; + ring_data[idx] = data[i]; + idx = (idx + 1) & ring_mask; } ring->tail = tail + num; diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index 8b9a70bbc..7e8b7e34d 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -687,7 +687,8 @@ static inline int _sched_queue_enq_multi(odp_queue_t handle, return -1; } - num_enq = ring_st_enq_multi(ring_st, buf_idx, num); + num_enq = ring_st_enq_multi(ring_st, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); if (odp_unlikely(num_enq == 0)) { UNLOCK(queue); @@ -728,7 +729,8 @@ int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int max_num, return -1; } - num_deq = ring_st_deq_multi(ring_st, buf_idx, max_num); + num_deq = ring_st_deq_multi(ring_st, queue->s.ring_data, + queue->s.ring_mask, buf_idx, max_num); if (num_deq == 0) { /* Already empty queue */ @@ -875,8 +877,10 @@ static int queue_init(queue_entry_t *queue, const char *name, } else { queue->s.enqueue = sched_queue_enq; queue->s.enqueue_multi = sched_queue_enq_multi; - ring_st_init(&queue->s.ring_st, - &queue_glb->ring_data[offset], queue_size); + + queue->s.ring_data = &queue_glb->ring_data[offset]; + queue->s.ring_mask = queue_size - 1; + ring_st_init(&queue->s.ring_st); } } From patchwork Fri Aug 24 14:00:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 145070 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1304832ljw; Fri, 24 Aug 2018 07:02:23 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaZobVQI5Zs1cNLbhpZ+SGrSgORS+hd6ukLX2raDfxZArLBwEEO8JuzzA+aumKBQTMWEHmA X-Received: by 2002:a0c:f312:: with SMTP id j18-v6mr1883607qvl.124.1535119343360; Fri, 24 Aug 2018 07:02:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535119343; cv=none; d=google.com; s=arc-20160816; b=KpMs5qBKNKDCrMVcUZM1qpovRPl0733Mhde3h82ZYDBQbypNvEQCu6hZ+fRBKtDl69 zDf8CZqjzeaeBJemaRyJH/pQH5fGq5bLQXJI0txJCgqoYxdDAjHgGSKX920QSNJZFN2I /MVg5LdrpoWCCX0TsaX1HfSuFZDAWpPoCKAQNe7JmrM9Q/BmS1pbzW6wE9YaKn9ErLzb I+MXq7evaIGrqa0ovmTDHtuFbINzlkETC6P3uMgw/5brCgCNFVkWfte1k9TMiI5YObBg /uHfLfqn+4gLmus5WtzbmMs3mf7oVXYLgzHiF2d7fgeXstVB0Mh+1B/3gM9h8sqFM92l Idqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=4+S0msG0aBI4pHZHYCX8JfAWaIxRatl8i9scj49rpn4=; b=RwslfzB7RQKYMW5kCsGQ2OVMYY/4jS6n11uYjRe4m27wFzxTE8AmzVCloAKmA3WjBI 4Cs59LK8oByHg5xuxKEj2vh5xjMnl5ZUQyn2qVR1ROx2/suvqXVIG9/5V94rLOWbmrMP jXHlLKRRUiHVGrp8RhoYllxi0pTZACz8fW65zkRJLeuUV5xDIY6maKsKD1yGJ9I9zzQS H3zlHNQDrvSzt1/JLf6X7PWm14KLwvM2oiW7tv3Beb2SY5on8tmyrPbgjBUaJaCr/cU9 LBzvp0W4rtWQ+bdtHNSznctZwzmp/JxGkG8Vdn5NXN+MVB0EtD4DTAs3RN/oZRSbfVQh MaLg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id h14-v6si178985qkj.259.2018.08.24.07.02.23; Fri, 24 Aug 2018 07:02:23 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 1414A685E3; Fri, 24 Aug 2018 14:02:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 6A681685FE; Fri, 24 Aug 2018 14:00:35 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 4363B685FE; Fri, 24 Aug 2018 14:00:21 +0000 (UTC) Received: from forward106p.mail.yandex.net (forward106p.mail.yandex.net [77.88.28.109]) by lists.linaro.org (Postfix) with ESMTPS id 1F0AE60833 for ; Fri, 24 Aug 2018 14:00:13 +0000 (UTC) Received: from mxback3o.mail.yandex.net (mxback3o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1d]) by forward106p.mail.yandex.net (Yandex) with ESMTP id ACC6C2D85F0E for ; Fri, 24 Aug 2018 17:00:11 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback3o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id Md1nAvcXtE-0BxSXWNp; Fri, 24 Aug 2018 17:00:11 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 2CVvDjLgK3-0AjGUcfh; Fri, 24 Aug 2018 17:00:10 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 24 Aug 2018 14:00:06 +0000 Message-Id: <1535119206-23556-5-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> References: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 683 Subject: [lng-odp] [PATCH v1 4/4] linux-gen: ring_spsc: move ring mask and data pointer X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Store mask and data pointer in queue entry instead of ring structure. Data is constant and can be stored among other frequently used read only data. Also other ring type use the same variables. Signed-off-by: Petri Savolainen --- /** Email created from pull request 683 (psavol:master-queue-lockless-enqdeq-3) ** https://github.com/Linaro/odp/pull/683 ** Patch: https://github.com/Linaro/odp/pull/683.patch ** Base sha: 989df5d2f97ab4711328b11282dcc743f5740e00 ** Merge commit sha: 28073c54671148efdd01c9cf38c1a235d5a133f0 **/ .../include/odp_ring_spsc_internal.h | 33 +++++++++---------- platform/linux-generic/odp_queue_spsc.c | 11 ++++--- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/platform/linux-generic/include/odp_ring_spsc_internal.h b/platform/linux-generic/include/odp_ring_spsc_internal.h index e38bda1da..de122bf57 100644 --- a/platform/linux-generic/include/odp_ring_spsc_internal.h +++ b/platform/linux-generic/include/odp_ring_spsc_internal.h @@ -29,31 +29,27 @@ extern "C" { typedef struct { odp_atomic_u32_t head; odp_atomic_u32_t tail; - uint32_t mask; - uint32_t *data; } ring_spsc_t; /* Initialize ring. Ring size must be a power of two. */ -static inline void ring_spsc_init(ring_spsc_t *ring, uint32_t *data, - uint32_t size) +static inline void ring_spsc_init(ring_spsc_t *ring) { odp_atomic_init_u32(&ring->head, 0); odp_atomic_init_u32(&ring->tail, 0); - ring->mask = size - 1; - ring->data = data; } /* Dequeue data from the ring head. Max_num is smaller than ring size.*/ -static inline uint32_t ring_spsc_deq_multi(ring_spsc_t *ring, uint32_t data[], +static inline uint32_t ring_spsc_deq_multi(ring_spsc_t *ring, + uint32_t *ring_data, + uint32_t ring_mask, uint32_t data[], uint32_t max_num) { - uint32_t head, tail, mask, idx; + uint32_t head, tail, idx; uint32_t num, i; tail = odp_atomic_load_acq_u32(&ring->tail); head = odp_atomic_load_u32(&ring->head); - mask = ring->mask; num = tail - head; /* Empty */ @@ -63,11 +59,11 @@ static inline uint32_t ring_spsc_deq_multi(ring_spsc_t *ring, uint32_t data[], if (num > max_num) num = max_num; - idx = head & mask; + idx = head & ring_mask; for (i = 0; i < num; i++) { - data[i] = ring->data[idx]; - idx = (idx + 1) & mask; + data[i] = ring_data[idx]; + idx = (idx + 1) & ring_mask; } odp_atomic_store_rel_u32(&ring->head, head + num); @@ -77,16 +73,17 @@ static inline uint32_t ring_spsc_deq_multi(ring_spsc_t *ring, uint32_t data[], /* Enqueue data into the ring tail. Num_data is smaller than ring size. */ static inline uint32_t ring_spsc_enq_multi(ring_spsc_t *ring, + uint32_t *ring_data, + uint32_t ring_mask, const uint32_t data[], uint32_t num_data) { - uint32_t head, tail, mask, size, idx; + uint32_t head, tail, size, idx; uint32_t num, i; head = odp_atomic_load_acq_u32(&ring->head); tail = odp_atomic_load_u32(&ring->tail); - mask = ring->mask; - size = mask + 1; + size = ring_mask + 1; num = size - (tail - head); /* Full */ @@ -96,11 +93,11 @@ static inline uint32_t ring_spsc_enq_multi(ring_spsc_t *ring, if (num > num_data) num = num_data; - idx = tail & mask; + idx = tail & ring_mask; for (i = 0; i < num; i++) { - ring->data[idx] = data[i]; - idx = (idx + 1) & mask; + ring_data[idx] = data[i]; + idx = (idx + 1) & ring_mask; } odp_atomic_store_rel_u32(&ring->tail, tail + num); diff --git a/platform/linux-generic/odp_queue_spsc.c b/platform/linux-generic/odp_queue_spsc.c index 0fd8d85a7..002561a49 100644 --- a/platform/linux-generic/odp_queue_spsc.c +++ b/platform/linux-generic/odp_queue_spsc.c @@ -49,7 +49,8 @@ static inline int spsc_enq_multi(odp_queue_t handle, return -1; } - return ring_spsc_enq_multi(ring_spsc, buf_idx, num); + return ring_spsc_enq_multi(ring_spsc, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); } static inline int spsc_deq_multi(odp_queue_t handle, @@ -68,7 +69,8 @@ static inline int spsc_deq_multi(odp_queue_t handle, return -1; } - num_deq = ring_spsc_deq_multi(ring_spsc, buf_idx, num); + num_deq = ring_spsc_deq_multi(ring_spsc, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); if (num_deq == 0) return 0; @@ -127,6 +129,7 @@ void queue_spsc_init(queue_entry_t *queue, uint32_t queue_size) offset = queue->s.index * (uint64_t)queue_glb->config.max_queue_size; - ring_spsc_init(&queue->s.ring_spsc, &queue_glb->ring_data[offset], - queue_size); + queue->s.ring_data = &queue_glb->ring_data[offset]; + queue->s.ring_mask = queue_size - 1; + ring_spsc_init(&queue->s.ring_spsc); }