From patchwork Fri Aug 24 14:00:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 145067 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1302728ljw; Fri, 24 Aug 2018 07:00:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZH0/pXKbnw5LNR+XjqN54BRCqnEpG6Xr1YGG69Rksd7Mmtnmrlwc3lDVspg8Urj0Yicmw1 X-Received: by 2002:ac8:2463:: with SMTP id d32-v6mr1832114qtd.41.1535119252822; Fri, 24 Aug 2018 07:00:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535119252; cv=none; d=google.com; s=arc-20160816; b=B0ez0fmFgpmEwSriGRMmqH30bP8w6f8/zNbIwVdkt0zME69u/9qGT79jlVXQnvNrhZ B/v3GJmtChzRXckGIVTt3g3hrypZ7EPpCaL92xF5Ka5RbjG9DJFgj5Odsy//HHp/T19I 3lljMsqC7wc1xeUcvzeLjb4bgjbv77jQWIyrAZ3GJCn+M7tDah7EAtclm+EbEIVuvd50 6oc0gwjFCDiTxAF9KJtx73FwMD7oD8R7BgBB6Ia0pqfZ5lMs9oT3b5yBjtzT/DwcWWkB xPPCH4w9zDKUQi/qOSot0leRRGaRdCiOsqSe47xOBGrcZ6kTp39CBXMF2IAJ6wdQcClG wN4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=VJSHsIQMSlq1OCS6u5z+CUzH3Dmtj0OboN9p132fa6Q=; b=S6SCerBsKSzHCHLKRc/BYNJdnNXiIqqsBdM0HyXkY7gR3QYi7zqzXc59iowfb3AuK7 B5pWjZziyYGAm5H6vFtV63YYBtXe3jYV9mgle0nyyk0OV0fz6v4Pjgp/UXUFU/otaSab y+SQkbdREtP8jowh9+WY/+Xs1cKQvtQli7jcMiMzdQakD+qlyQqx8MY1I4/0ZDWLe8Nx MzoIVntU9Y1rhODValvJ1PuY9NU2u/Yr1Rf8xfGoHH313HgolLYTdrXDdXnqtfCpMYmj soz4XRjzlnHZ+G2samUbIfsPPhA43LcUclj7L8kNACtlM+TPqzmvpIsjI9k4pI8AQOCG JPRw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id p1-v6si150390qta.346.2018.08.24.07.00.52; Fri, 24 Aug 2018 07:00:52 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 68D92608B7; Fri, 24 Aug 2018 14:00:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 5D7E260800; Fri, 24 Aug 2018 14:00:20 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 75E3F608B7; Fri, 24 Aug 2018 14:00:13 +0000 (UTC) Received: from forward102o.mail.yandex.net (forward102o.mail.yandex.net [37.140.190.182]) by lists.linaro.org (Postfix) with ESMTPS id 80332607C8 for ; Fri, 24 Aug 2018 14:00:11 +0000 (UTC) Received: from mxback6j.mail.yandex.net (mxback6j.mail.yandex.net [IPv6:2a02:6b8:0:1619::10f]) by forward102o.mail.yandex.net (Yandex) with ESMTP id 1BF045A04177 for ; Fri, 24 Aug 2018 17:00:10 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback6j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id ZM1x0AU2Q7-0ArCQCZw; Fri, 24 Aug 2018 17:00:10 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 2CVvDjLgK3-09jeHamU; Fri, 24 Aug 2018 17:00:09 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 24 Aug 2018 14:00:04 +0000 Message-Id: <1535119206-23556-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> References: <1535119206-23556-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 683 Subject: [lng-odp] [PATCH v1 2/4] linux-gen: queue: use mpmc ring in plain queues X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Change plain queue implementation to use ring_mpmc instead of ticket lock and ring_st ring. Performance and scalability improves especially on 64 bit ARM. Signed-off-by: Petri Savolainen --- /** Email created from pull request 683 (psavol:master-queue-lockless-enqdeq-3) ** https://github.com/Linaro/odp/pull/683 ** Patch: https://github.com/Linaro/odp/pull/683.patch ** Base sha: 989df5d2f97ab4711328b11282dcc743f5740e00 ** Merge commit sha: 28073c54671148efdd01c9cf38c1a235d5a133f0 **/ .../include/odp_queue_basic_internal.h | 30 +++++++----- platform/linux-generic/odp_queue_basic.c | 48 +++++++------------ 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/platform/linux-generic/include/odp_queue_basic_internal.h b/platform/linux-generic/include/odp_queue_basic_internal.h index 15e49772c..46b747955 100644 --- a/platform/linux-generic/include/odp_queue_basic_internal.h +++ b/platform/linux-generic/include/odp_queue_basic_internal.h @@ -22,6 +22,7 @@ extern "C" { #include #include #include +#include #include #include #include @@ -33,22 +34,29 @@ extern "C" { #define QUEUE_STATUS_SCHED 4 struct queue_entry_s { - odp_ticketlock_t ODP_ALIGNED_CACHE lock; - union { - ring_st_t ring_st; - ring_spsc_t ring_spsc; - }; - int status; - + /* The first cache line is read only */ queue_enq_fn_t ODP_ALIGNED_CACHE enqueue; queue_deq_fn_t dequeue; queue_enq_multi_fn_t enqueue_multi; queue_deq_multi_fn_t dequeue_multi; - queue_deq_multi_fn_t orig_dequeue_multi; + uint32_t *ring_data; + uint32_t ring_mask; + uint32_t index; + odp_queue_t handle; + odp_queue_type_t type; + + /* MPMC ring (2 cache lines). */ + ring_mpmc_t ring_mpmc; - uint32_t index; - odp_queue_t handle; - odp_queue_type_t type; + odp_ticketlock_t lock; + union { + ring_st_t ring_st; + ring_spsc_t ring_spsc; + }; + + int status; + + queue_deq_multi_fn_t orig_dequeue_multi; odp_queue_param_t param; odp_pktin_queue_t pktin; odp_pktout_queue_t pktout; diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index e5d915643..8b9a70bbc 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -400,8 +400,10 @@ static int queue_destroy(odp_queue_t handle) if (queue->s.spsc) empty = ring_spsc_is_empty(&queue->s.ring_spsc); - else + else if (queue->s.type == ODP_QUEUE_TYPE_SCHED) empty = ring_st_is_empty(&queue->s.ring_st); + else + empty = ring_mpmc_is_empty(&queue->s.ring_mpmc); if (!empty) { UNLOCK(queue); @@ -490,28 +492,19 @@ static inline int _plain_queue_enq_multi(odp_queue_t handle, { queue_entry_t *queue; int ret, num_enq; - ring_st_t *ring_st; + ring_mpmc_t *ring_mpmc; uint32_t buf_idx[num]; queue = qentry_from_handle(handle); - ring_st = &queue->s.ring_st; + ring_mpmc = &queue->s.ring_mpmc; if (sched_fn->ord_enq_multi(handle, (void **)buf_hdr, num, &ret)) return ret; buffer_index_from_buf(buf_idx, buf_hdr, num); - LOCK(queue); - - if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { - UNLOCK(queue); - ODP_ERR("Bad queue status\n"); - return -1; - } - - num_enq = ring_st_enq_multi(ring_st, buf_idx, num); - - UNLOCK(queue); + num_enq = ring_mpmc_enq_multi(ring_mpmc, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); return num_enq; } @@ -521,23 +514,14 @@ static inline int _plain_queue_deq_multi(odp_queue_t handle, { int num_deq; queue_entry_t *queue; - ring_st_t *ring_st; + ring_mpmc_t *ring_mpmc; uint32_t buf_idx[num]; queue = qentry_from_handle(handle); - ring_st = &queue->s.ring_st; + ring_mpmc = &queue->s.ring_mpmc; - LOCK(queue); - - if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { - /* Bad queue, or queue has been destroyed. */ - UNLOCK(queue); - return -1; - } - - num_deq = ring_st_deq_multi(ring_st, buf_idx, num); - - UNLOCK(queue); + num_deq = ring_mpmc_deq_multi(ring_mpmc, queue->s.ring_data, + queue->s.ring_mask, buf_idx, num); if (num_deq == 0) return 0; @@ -883,13 +867,17 @@ static int queue_init(queue_entry_t *queue, const char *name, queue->s.dequeue = plain_queue_deq; queue->s.dequeue_multi = plain_queue_deq_multi; queue->s.orig_dequeue_multi = plain_queue_deq_multi; + + queue->s.ring_data = &queue_glb->ring_data[offset]; + queue->s.ring_mask = queue_size - 1; + ring_mpmc_init(&queue->s.ring_mpmc); + } else { queue->s.enqueue = sched_queue_enq; queue->s.enqueue_multi = sched_queue_enq_multi; + ring_st_init(&queue->s.ring_st, + &queue_glb->ring_data[offset], queue_size); } - - ring_st_init(&queue->s.ring_st, &queue_glb->ring_data[offset], - queue_size); } return 0;