From patchwork Sun May 17 19:57:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 219084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31951C433E1 for ; Sun, 17 May 2020 19:57:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D76520758 for ; Sun, 17 May 2020 19:57:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="ZGN3upZe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726550AbgEQT5r (ORCPT ); Sun, 17 May 2020 15:57:47 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:17210 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726269AbgEQT5q (ORCPT ); Sun, 17 May 2020 15:57:46 -0400 Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 04HJtSMS020898 for ; Sun, 17 May 2020 12:57:45 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=HisIGKAv4VrnMJx7LxmQhcH6r5aSNlVCdz642a7/snU=; b=ZGN3upZesJEg1pHFQ2+tB1eULEOu5b7hI7sViyLuSSNuV4bRYSlDsuoxU5z3F9gqiXb2 VjtRZObJqvUW+09v1jywK5nf1l36/0mURxw5Tar2xSmAk0ANQ6pSRkW6HUunuEjuDEmF S/GLVjhJAI0wRyJvITvvPx8vaBVIN4nc3+k= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 3130134eyx-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sun, 17 May 2020 12:57:45 -0700 Received: from intmgw005.03.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1847.3; Sun, 17 May 2020 12:57:44 -0700 Received: by devbig012.ftw2.facebook.com (Postfix, from userid 137359) id 7B7FB2EC32DC; Sun, 17 May 2020 12:57:39 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Andrii Nakryiko Smtp-Origin-Hostname: devbig012.ftw2.facebook.com To: , , , CC: , , Andrii Nakryiko , "Paul E . McKenney" , Jonathan Lemon Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v2 bpf-next 2/7] tools/memory-model: add BPF ringbuf MPSC litmus tests Date: Sun, 17 May 2020 12:57:22 -0700 Message-ID: <20200517195727.279322-3-andriin@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200517195727.279322-1-andriin@fb.com> References: <20200517195727.279322-1-andriin@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-17_07:2020-05-15,2020-05-17 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 bulkscore=0 phishscore=0 priorityscore=1501 suspectscore=8 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 impostorscore=0 mlxscore=0 malwarescore=0 mlxlogscore=387 cotscore=-2147483648 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2005170182 X-FB-Internal: deliver Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add 4 litmus tests for BPF ringbuf implementation, divided into two different use cases. First, two unbounded case, one with 1 producer and another with 2 producers, single consumer. All reservations are supposed to succeed. Second, bounded case with only 1 record allowed in ring buffer at any given time. Here failures to reserve space are expected. Again, 1- and 2- producer cases, single consumer, are validated. Just for the fun of it, I also wrote a 3-producer cases, it took *16 hours* to validate, but came back successful as well. I'm not including it in this patch, because it's not practical to run it. See output for all included 4 cases and one 3-producer one with bounded use case. Each litmust test implements producer/consumer protocol for BPF ring buffer implementation found in kernel/bpf/ringbuf.c. Due to limitations, all records are assumed equal-sized and producer/consumer counters are incremented by 1. This doesn't change the correctness of the algorithm, though. Verification results: /* 1p1c bounded case */ $ herd7 -unroll 0 -conf linux-kernel.cfg litmus-tests/mpsc-rb+1p1c+bounded.litmus Test mpsc-rb+1p1c+bounded Allowed States 2 0:rFail=0; 1:rFail=0; cx=0; dropped=0; len1=1; px=1; 0:rFail=0; 1:rFail=0; cx=1; dropped=0; len1=1; px=1; Ok Witnesses Positive: 3 Negative: 0 Condition exists (0:rFail=0 /\ 1:rFail=0 /\ dropped=0 /\ px=1 /\ len1=1 /\ (cx=0 \/ cx=1)) Observation mpsc-rb+1p1c+bounded Always 3 0 Time mpsc-rb+1p1c+bounded 0.03 Hash=5bdad0f41557a641370e7fa6b8eb2f43 /* 2p1c bounded case */ $ herd7 -unroll 0 -conf linux-kernel.cfg litmus-tests/mpsc-rb+2p1c+bounded.litmus Test mpsc-rb+2p1c+bounded Allowed States 4 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=0; dropped=1; len1=1; px=1; 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=1; dropped=0; len1=1; px=2; 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=1; dropped=1; len1=1; px=1; 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=2; dropped=0; len1=1; px=2; Ok Witnesses Positive: 22 Negative: 0 Condition exists (0:rFail=0 /\ 1:rFail=0 /\ 2:rFail=0 /\ len1=1 /\ (dropped=0 /\ px=2 /\ (cx=1 \/ cx=2) \/ dropped=1 /\ px=1 /\ (cx=0 \/ cx=1))) Observation mpsc-rb+2p1c+bounded Always 22 0 Time mpsc-rb+2p1c+bounded 119.38 Hash=e2f8f442a02bf7d8c2988ba82cf002d2 /* 1p1c unbounded case */ $ herd7 -unroll 0 -conf linux-kernel.cfg litmus-tests/mpsc-rb+1p1c+unbound.litmus Test mpsc-rb+1p1c+unbound Allowed States 2 0:rFail=0; 1:rFail=0; cx=0; len1=1; px=1; 0:rFail=0; 1:rFail=0; cx=1; len1=1; px=1; Ok Witnesses Positive: 3 Negative: 0 Condition exists (0:rFail=0 /\ 1:rFail=0 /\ px=1 /\ len1=1 /\ (cx=0 \/ cx=1)) Observation mpsc-rb+1p1c+unbound Always 3 0 Time mpsc-rb+1p1c+unbound 0.02 Hash=be9de6487d8e27c3d37802d122e4a87c /* 2p1c unbounded case */ $ herd7 -unroll 0 -conf linux-kernel.cfg litmus-tests/mpsc-rb+2p1c+unbound.litmus Test mpsc-rb+2p1c+unbound Allowed States 3 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=0; len1=1; len2=1; px=2; 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=1; len1=1; len2=1; px=2; 0:rFail=0; 1:rFail=0; 2:rFail=0; cx=2; len1=1; len2=1; px=2; Ok Witnesses Positive: 42 Negative: 0 Condition exists (0:rFail=0 /\ 1:rFail=0 /\ 2:rFail=0 /\ px=2 /\ len1=1 /\ len2=1 /\ (cx=0 \/ cx=1 \/ cx=2)) Observation mpsc-rb+2p1c+unbound Always 42 0 Time mpsc-rb+2p1c+unbound 39.19 Hash=f0352aba9bdc03dd0b1def7d0c4956fa /* 3p1c bounded case */ $ herd7 -unroll 0 -conf linux-kernel.cfg mpsc-rb+3p1c+bounded.litmus Test mpsc+ringbuf-spinlock Allowed States 5 0:rFail=0; 1:rFail=0; 2:rFail=0; 3:rFail=0; cx=0; len1=1; len2=1; px=2; 0:rFail=0; 1:rFail=0; 2:rFail=0; 3:rFail=0; cx=1; len1=1; len2=1; px=2; 0:rFail=0; 1:rFail=0; 2:rFail=0; 3:rFail=0; cx=1; len1=1; len2=1; px=3; 0:rFail=0; 1:rFail=0; 2:rFail=0; 3:rFail=0; cx=2; len1=1; len2=1; px=2; 0:rFail=0; 1:rFail=0; 2:rFail=0; 3:rFail=0; cx=2; len1=1; len2=1; px=3; Ok Witnesses Positive: 558 Negative: 0 Condition exists (0:rFail=0 /\ 1:rFail=0 /\ 2:rFail=0 /\ 3:rFail=0 /\ len1=1 /\ len2=1 /\ (px=2 /\ (cx=0 \/ cx=1 \/ cx=2) \/ px=3 /\ (cx=1 \/ cx=2))) Observation mpsc+ringbuf-spinlock Always 558 0 Time mpsc+ringbuf-spinlock 57487.24 Hash=133977dba930d167b4e1b4a6923d5687 Cc: Paul E. McKenney Signed-off-by: Andrii Nakryiko --- .../litmus-tests/mpsc-rb+1p1c+bounded.litmus | 92 +++++++++++ .../litmus-tests/mpsc-rb+1p1c+unbound.litmus | 83 ++++++++++ .../litmus-tests/mpsc-rb+2p1c+bounded.litmus | 152 ++++++++++++++++++ .../litmus-tests/mpsc-rb+2p1c+unbound.litmus | 137 ++++++++++++++++ 4 files changed, 464 insertions(+) create mode 100644 tools/memory-model/litmus-tests/mpsc-rb+1p1c+bounded.litmus create mode 100644 tools/memory-model/litmus-tests/mpsc-rb+1p1c+unbound.litmus create mode 100644 tools/memory-model/litmus-tests/mpsc-rb+2p1c+bounded.litmus create mode 100644 tools/memory-model/litmus-tests/mpsc-rb+2p1c+unbound.litmus diff --git a/tools/memory-model/litmus-tests/mpsc-rb+1p1c+bounded.litmus b/tools/memory-model/litmus-tests/mpsc-rb+1p1c+bounded.litmus new file mode 100644 index 000000000000..cafd17afe11e --- /dev/null +++ b/tools/memory-model/litmus-tests/mpsc-rb+1p1c+bounded.litmus @@ -0,0 +1,92 @@ +C mpsc-rb+1p1c+bounded + +(* + * Result: Always + * + * This litmus test validates BPF ring buffer implementation under the + * following assumptions: + * - 1 producer; + * - 1 consumer; + * - ring buffer has capacity for only 1 record. + * + * Expectations: + * - 1 record pushed into ring buffer; + * - 0 or 1 element is consumed. + * - no failures. + *) + +{ + max_len = 1; + len1 = 0; + px = 0; + cx = 0; + dropped = 0; +} + +P0(int *len1, int *cx, int *px) +{ + int *rLenPtr; + int rLen; + int rPx; + int rCx; + int rFail; + + rFail = 0; + rCx = smp_load_acquire(cx); + + rPx = smp_load_acquire(px); + if (rCx < rPx) { + if (rCx == 0) + rLenPtr = len1; + else + rFail = 1; + + rLen = smp_load_acquire(rLenPtr); + if (rLen == 0) { + rFail = 1; + } else if (rLen == 1) { + rCx = rCx + 1; + smp_store_release(cx, rCx); + } + } +} + +P1(int *len1, spinlock_t *rb_lock, int *px, int *cx, int *dropped, int *max_len) +{ + int rPx; + int rCx; + int rFail; + int *rLenPtr; + + rFail = 0; + rCx = smp_load_acquire(cx); + + spin_lock(rb_lock); + + rPx = *px; + if (rPx - rCx >= *max_len) { + atomic_inc(dropped); + spin_unlock(rb_lock); + } else { + if (rPx == 0) + rLenPtr = len1; + else + rFail = 1; + + *rLenPtr = -1; + smp_wmb(); + smp_store_release(px, rPx + 1); + + spin_unlock(rb_lock); + + smp_store_release(rLenPtr, 1); + } +} + +exists ( + 0:rFail=0 /\ 1:rFail=0 + /\ + ( + (dropped=0 /\ px=1 /\ len1=1 /\ (cx=0 \/ cx=1)) + ) +) diff --git a/tools/memory-model/litmus-tests/mpsc-rb+1p1c+unbound.litmus b/tools/memory-model/litmus-tests/mpsc-rb+1p1c+unbound.litmus new file mode 100644 index 000000000000..84f660598015 --- /dev/null +++ b/tools/memory-model/litmus-tests/mpsc-rb+1p1c+unbound.litmus @@ -0,0 +1,83 @@ +C mpsc-rb+1p1c+unbound + +(* + * Result: Always + * + * This litmus test validates BPF ring buffer implementation under the + * following assumptions: + * - 1 producer; + * - 1 consumer; + * - ring buffer capacity is unbounded. + * + * Expectations: + * - 1 record pushed into ring buffer; + * - 0 or 1 element is consumed. + * - no failures. + *) + +{ + len1 = 0; + px = 0; + cx = 0; +} + +P0(int *len1, int *cx, int *px) +{ + int *rLenPtr; + int rLen; + int rPx; + int rCx; + int rFail; + + rFail = 0; + rCx = smp_load_acquire(cx); + + rPx = smp_load_acquire(px); + if (rCx < rPx) { + if (rCx == 0) + rLenPtr = len1; + else + rFail = 1; + + rLen = smp_load_acquire(rLenPtr); + if (rLen == 0) { + rFail = 1; + } else if (rLen == 1) { + rCx = rCx + 1; + smp_store_release(cx, rCx); + } + } +} + +P1(int *len1, spinlock_t *rb_lock, int *px, int *cx) +{ + int rPx; + int rCx; + int rFail; + int *rLenPtr; + + rFail = 0; + rCx = smp_load_acquire(cx); + + spin_lock(rb_lock); + + rPx = *px; + if (rPx == 0) + rLenPtr = len1; + else + rFail = 1; + + *rLenPtr = -1; + smp_wmb(); + smp_store_release(px, rPx + 1); + + spin_unlock(rb_lock); + + smp_store_release(rLenPtr, 1); +} + +exists ( + 0:rFail=0 /\ 1:rFail=0 + /\ px=1 /\ len1=1 + /\ (cx=0 \/ cx=1) +) diff --git a/tools/memory-model/litmus-tests/mpsc-rb+2p1c+bounded.litmus b/tools/memory-model/litmus-tests/mpsc-rb+2p1c+bounded.litmus new file mode 100644 index 000000000000..900104c4933b --- /dev/null +++ b/tools/memory-model/litmus-tests/mpsc-rb+2p1c+bounded.litmus @@ -0,0 +1,152 @@ +C mpsc-rb+2p1c+bounded + +(* + * Result: Always + * + * This litmus test validates BPF ring buffer implementation under the + * following assumptions: + * - 2 identical producers; + * - 1 consumer; + * - ring buffer has capacity for only 1 record. + * + * Expectations: + * - either 1 or 2 records are pushed into ring buffer; + * - 0, 1, or 2 elements are consumed by consumer; + * - appropriate number of dropped records is recorded to satisfy ring buffer + * size bounds; + * - no failures. + *) + +{ + max_len = 1; + len1 = 0; + px = 0; + cx = 0; + dropped = 0; +} + +P0(int *len1, int *cx, int *px) +{ + int *rLenPtr; + int rLen; + int rPx; + int rCx; + int rFail; + + rFail = 0; + rCx = smp_load_acquire(cx); + + rPx = smp_load_acquire(px); + if (rCx < rPx) { + if (rCx == 0) + rLenPtr = len1; + else if (rCx == 1) + rLenPtr = len1; + else + rFail = 1; + + rLen = smp_load_acquire(rLenPtr); + if (rLen == 0) { + rFail = 1; + } else if (rLen == 1) { + rCx = rCx + 1; + smp_store_release(cx, rCx); + } + } + + rPx = smp_load_acquire(px); + if (rCx < rPx) { + if (rCx == 0) + rLenPtr = len1; + else if (rCx == 1) + rLenPtr = len1; + else + rFail = 1; + + rLen = smp_load_acquire(rLenPtr); + if (rLen == 0) { + rFail = 1; + } else if (rLen == 1) { + rCx = rCx + 1; + smp_store_release(cx, rCx); + } + } +} + +P1(int *len1, spinlock_t *rb_lock, int *px, int *cx, int *dropped, int *max_len) +{ + int rPx; + int rCx; + int rFail; + int *rLenPtr; + + rFail = 0; + rCx = smp_load_acquire(cx); + + spin_lock(rb_lock); + + rPx = *px; + if (rPx - rCx >= *max_len) { + atomic_inc(dropped); + spin_unlock(rb_lock); + } else { + if (rPx == 0) + rLenPtr = len1; + else if (rPx == 1) + rLenPtr = len1; + else + rFail = 1; + + *rLenPtr = -1; + smp_wmb(); + smp_store_release(px, rPx + 1); + + spin_unlock(rb_lock); + + smp_store_release(rLenPtr, 1); + } +} + +P2(int *len1, spinlock_t *rb_lock, int *px, int *cx, int *dropped, int *max_len) +{ + int rPx; + int rCx; + int rFail; + int *rLenPtr; + + rFail = 0; + rCx = smp_load_acquire(cx); + + spin_lock(rb_lock); + + rPx = *px; + if (rPx - rCx >= *max_len) { + atomic_inc(dropped); + spin_unlock(rb_lock); + } else { + if (rPx == 0) + rLenPtr = len1; + else if (rPx == 1) + rLenPtr = len1; + else + rFail = 1; + + *rLenPtr = -1; + smp_wmb(); + smp_store_release(px, rPx + 1); + + spin_unlock(rb_lock); + + smp_store_release(rLenPtr, 1); + } +} + +exists ( + 0:rFail=0 /\ 1:rFail=0 /\ 2:rFail=0 /\ len1=1 + /\ + ( + (dropped = 0 /\ px=2 /\ (cx=1 \/ cx=2)) + \/ + (dropped = 1 /\ px=1 /\ (cx=0 \/ cx=1)) + ) +) diff --git a/tools/memory-model/litmus-tests/mpsc-rb+2p1c+unbound.litmus b/tools/memory-model/litmus-tests/mpsc-rb+2p1c+unbound.litmus new file mode 100644 index 000000000000..83372e9eb079 --- /dev/null +++ b/tools/memory-model/litmus-tests/mpsc-rb+2p1c+unbound.litmus @@ -0,0 +1,137 @@ +C mpsc-rb+2p1c+unbound + +(* + * Result: Always + * + * This litmus test validates BPF ring buffer implementation under the + * following assumptions: + * - 2 identical producers; + * - 1 consumer; + * - ring buffer capacity is unbounded. + * + * Expectations: + * - 2 records pushed into ring buffer; + * - 0, 1, or 2 elements are consumed. + * - no failures. + *) + +{ + len1 = 0; + len2 = 0; + px = 0; + cx = 0; +} + +P0(int *len1, int *len2, int *cx, int *px) +{ + int *rLenPtr; + int rLen; + int rPx; + int rCx; + int rFail; + + rFail = 0; + rCx = smp_load_acquire(cx); + + rPx = smp_load_acquire(px); + if (rCx < rPx) { + if (rCx == 0) + rLenPtr = len1; + else if (rCx == 1) + rLenPtr = len2; + else + rFail = 1; + + rLen = smp_load_acquire(rLenPtr); + if (rLen == 0) { + rFail = 1; + } else if (rLen == 1) { + rCx = rCx + 1; + smp_store_release(cx, rCx); + } + } + + rPx = smp_load_acquire(px); + if (rCx < rPx) { + if (rCx == 0) + rLenPtr = len1; + else if (rCx == 1) + rLenPtr = len2; + else + rFail = 1; + + rLen = smp_load_acquire(rLenPtr); + if (rLen == 0) { + rFail = 1; + } else if (rLen == 1) { + rCx = rCx + 1; + smp_store_release(cx, rCx); + } + } +} + +P1(int *len1, int *len2, spinlock_t *rb_lock, int *px, int *cx) +{ + int rPx; + int rCx; + int rFail; + int *rLenPtr; + + rFail = 0; + rCx = smp_load_acquire(cx); + + spin_lock(rb_lock); + + rPx = *px; + if (rPx == 0) + rLenPtr = len1; + else if (rPx == 1) + rLenPtr = len2; + else + rFail = 1; + + *rLenPtr = -1; + smp_wmb(); + smp_store_release(px, rPx + 1); + + spin_unlock(rb_lock); + + smp_store_release(rLenPtr, 1); +} + +P2(int *len1, int *len2, spinlock_t *rb_lock, int *px, int *cx) +{ + int rPx; + int rCx; + int rFail; + int *rLenPtr; + + rFail = 0; + rCx = smp_load_acquire(cx); + + spin_lock(rb_lock); + + rPx = *px; + if (rPx == 0) + rLenPtr = len1; + else if (rPx == 1) + rLenPtr = len2; + else + rFail = 1; + + *rLenPtr = -1; + smp_wmb(); + smp_store_release(px, rPx + 1); + + spin_unlock(rb_lock); + + smp_store_release(rLenPtr, 1); +} + +exists ( + 0:rFail=0 /\ 1:rFail=0 /\ 2:rFail=0 + /\ + px=2 /\ len1=1 /\ len2=1 + /\ + (cx=0 \/ cx=1 \/ cx=2) +) From patchwork Sun May 17 19:57:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 219082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A61F9C433E0 for ; Sun, 17 May 2020 19:57:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 83DF520758 for ; Sun, 17 May 2020 19:57:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="l/WFNd1J" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726731AbgEQT55 (ORCPT ); Sun, 17 May 2020 15:57:57 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:17308 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726379AbgEQT5v (ORCPT ); Sun, 17 May 2020 15:57:51 -0400 Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.42/8.16.0.42) with SMTP id 04HJskxP010710 for ; Sun, 17 May 2020 12:57:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=jEXE0vLclit2NksU7oBOAvCzoWIqrN+Pu/L3fEeO+Ug=; b=l/WFNd1JirzQk9iK+9sTuiBTr5VOtD0SHl+J3a0FOBz1xzBVCcVjwFkMRUoLgTNRfcV1 2Ol3uo+4zTwm0tPk5q2hD+72cSZIfrH5yEbuViZhnpmVjYoaOuhh7tPfHH/IuJ6jeTTT jNKB68PWwN6lPuz6mncGYOHrhBBCrGa+Bwg= Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net with ESMTP id 312bp0uknu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sun, 17 May 2020 12:57:49 -0700 Received: from intmgw001.03.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1847.3; Sun, 17 May 2020 12:57:48 -0700 Received: by devbig012.ftw2.facebook.com (Postfix, from userid 137359) id B5D792EC32DC; Sun, 17 May 2020 12:57:41 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Andrii Nakryiko Smtp-Origin-Hostname: devbig012.ftw2.facebook.com To: , , , CC: , , Andrii Nakryiko , "Paul E . McKenney" , Jonathan Lemon Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v2 bpf-next 3/7] bpf: track reference type in verifier Date: Sun, 17 May 2020 12:57:23 -0700 Message-ID: <20200517195727.279322-4-andriin@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200517195727.279322-1-andriin@fb.com> References: <20200517195727.279322-1-andriin@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-17_07:2020-05-15,2020-05-17 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 phishscore=0 priorityscore=1501 mlxlogscore=999 impostorscore=0 cotscore=-2147483648 malwarescore=0 suspectscore=25 bulkscore=0 mlxscore=0 clxscore=1015 spamscore=0 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2005170182 X-FB-Internal: deliver Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch extends verifier's reference tracking logic with tracking a reference type and ensuring acquire()/release() functions accept only the right types of references. Currently, ambiguity between socket and ringbuf record references is resolved through use of different types of input arguments to bpf_sk_release() and bpf_ringbuf_commit(): ARG_PTR_TO_SOCK_COMMON and ARG_PTR_TO_ALLOC_MEM, respectively. It is thus impossible to pass ringbuf record pointer to bpf_sk_release() (and vice versa for socket). On the other hand, patch #1 added ARG_PTR_TO_ALLOC_MEM arg type, which, from the point of view of verifier, is a pointer to a fixed-sized allocated memory region. This is generic enough concept that could be used for other BPF helpers (e.g., malloc/free pair, if added). once we have that, there will be nothing to prevent passing ringbuf record to bpf_mem_free() (or whatever) helper. To that end, this patch adds a capability to specify and track reference types, that would be validated by verifier to ensure correct match between acquire() and release() helpers. This patch can be postponed for later, so is posted separate from other verifier changes. Signed-off-by: Andrii Nakryiko --- include/linux/bpf_verifier.h | 8 +++ kernel/bpf/verifier.c | 96 +++++++++++++++++++++++++++++------- 2 files changed, 86 insertions(+), 18 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index ca08db4ffb5f..26b2c4730f5a 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -164,6 +164,12 @@ struct bpf_stack_state { u8 slot_type[BPF_REG_SIZE]; }; +enum bpf_ref_type { + BPF_REF_INVALID, + BPF_REF_SOCKET, + BPF_REF_RINGBUF, +}; + struct bpf_reference_state { /* Track each reference created with a unique id, even if the same * instruction creates the reference multiple times (eg, via CALL). @@ -173,6 +179,8 @@ struct bpf_reference_state { * is used purely to inform the user of a reference leak. */ int insn_idx; + /* Type of reference being tracked */ + enum bpf_ref_type ref_type; }; /* state of the program: diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4ce3e031554d..2a6a48c1e10b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -436,6 +436,19 @@ static bool is_release_function(enum bpf_func_id func_id) func_id == BPF_FUNC_ringbuf_discard; } +static enum bpf_ref_type get_release_ref_type(enum bpf_func_id func_id) +{ + switch (func_id) { + case BPF_FUNC_sk_release: + return BPF_REF_SOCKET; + case BPF_FUNC_ringbuf_submit: + case BPF_FUNC_ringbuf_discard: + return BPF_REF_RINGBUF; + default: + return BPF_REF_INVALID; + } +} + static bool may_be_acquire_function(enum bpf_func_id func_id) { return func_id == BPF_FUNC_sk_lookup_tcp || @@ -464,6 +477,28 @@ static bool is_acquire_function(enum bpf_func_id func_id, return false; } +static enum bpf_ref_type get_acquire_ref_type(enum bpf_func_id func_id, + const struct bpf_map *map) +{ + enum bpf_map_type map_type = map ? map->map_type : BPF_MAP_TYPE_UNSPEC; + + switch (func_id) { + case BPF_FUNC_sk_lookup_tcp: + case BPF_FUNC_sk_lookup_udp: + case BPF_FUNC_skc_lookup_tcp: + return BPF_REF_SOCKET; + case BPF_FUNC_map_lookup_elem: + if (map_type == BPF_MAP_TYPE_SOCKMAP || + map_type == BPF_MAP_TYPE_SOCKHASH) + return BPF_REF_SOCKET; + return BPF_REF_INVALID; + case BPF_FUNC_ringbuf_reserve: + return BPF_REF_RINGBUF; + default: + return BPF_REF_INVALID; + } +} + static bool is_ptr_cast_function(enum bpf_func_id func_id) { return func_id == BPF_FUNC_tcp_sock || @@ -736,7 +771,8 @@ static int realloc_func_state(struct bpf_func_state *state, int stack_size, * On success, returns a valid pointer id to associate with the register * On failure, returns a negative errno. */ -static int acquire_reference_state(struct bpf_verifier_env *env, int insn_idx) +static int acquire_reference_state(struct bpf_verifier_env *env, + int insn_idx, enum bpf_ref_type ref_type) { struct bpf_func_state *state = cur_func(env); int new_ofs = state->acquired_refs; @@ -748,25 +784,32 @@ static int acquire_reference_state(struct bpf_verifier_env *env, int insn_idx) id = ++env->id_gen; state->refs[new_ofs].id = id; state->refs[new_ofs].insn_idx = insn_idx; + state->refs[new_ofs].ref_type = ref_type; return id; } /* release function corresponding to acquire_reference_state(). Idempotent. */ -static int release_reference_state(struct bpf_func_state *state, int ptr_id) +static int release_reference_state(struct bpf_func_state *state, int ptr_id, + enum bpf_ref_type ref_type) { + struct bpf_reference_state *ref; int i, last_idx; last_idx = state->acquired_refs - 1; for (i = 0; i < state->acquired_refs; i++) { - if (state->refs[i].id == ptr_id) { - if (last_idx && i != last_idx) - memcpy(&state->refs[i], &state->refs[last_idx], - sizeof(*state->refs)); - memset(&state->refs[last_idx], 0, sizeof(*state->refs)); - state->acquired_refs--; - return 0; - } + ref = &state->refs[i]; + if (ref->id != ptr_id) + continue; + + if (ref_type != BPF_REF_INVALID && ref->ref_type != ref_type) + return -EINVAL; + + if (i != last_idx) + memcpy(ref, &state->refs[last_idx], sizeof(*ref)); + memset(&state->refs[last_idx], 0, sizeof(*ref)); + state->acquired_refs--; + return 0; } return -EINVAL; } @@ -4296,14 +4339,13 @@ static void release_reg_references(struct bpf_verifier_env *env, /* The pointer with the specified id has released its reference to kernel * resources. Identify all copies of the same pointer and clear the reference. */ -static int release_reference(struct bpf_verifier_env *env, - int ref_obj_id) +static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, + enum bpf_ref_type ref_type) { struct bpf_verifier_state *vstate = env->cur_state; - int err; - int i; + int err, i; - err = release_reference_state(cur_func(env), ref_obj_id); + err = release_reference_state(cur_func(env), ref_obj_id, ref_type); if (err) return err; @@ -4664,7 +4706,16 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn return err; } } else if (is_release_function(func_id)) { - err = release_reference(env, meta.ref_obj_id); + enum bpf_ref_type ref_type; + + ref_type = get_release_ref_type(func_id); + if (ref_type == BPF_REF_INVALID) { + verbose(env, "unrecognized reference accepted by func %s#%d\n", + func_id_name(func_id), func_id); + return -EFAULT; + } + + err = release_reference(env, meta.ref_obj_id, ref_type); if (err) { verbose(env, "func %s#%d reference has not been acquired before\n", func_id_name(func_id), func_id); @@ -4747,8 +4798,17 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn /* For release_reference() */ regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id; } else if (is_acquire_function(func_id, meta.map_ptr)) { - int id = acquire_reference_state(env, insn_idx); + enum bpf_ref_type ref_type; + int id; + + ref_type = get_acquire_ref_type(func_id, meta.map_ptr); + if (ref_type == BPF_REF_INVALID) { + verbose(env, "unrecognized reference returned by func %s#%d\n", + func_id_name(func_id), func_id); + return -EINVAL; + } + id = acquire_reference_state(env, insn_idx, ref_type); if (id < 0) return id; /* For mark_ptr_or_null_reg() */ @@ -6731,7 +6791,7 @@ static void mark_ptr_or_null_regs(struct bpf_verifier_state *vstate, u32 regno, * No one could have freed the reference state before * doing the NULL check. */ - WARN_ON_ONCE(release_reference_state(state, id)); + WARN_ON_ONCE(release_reference_state(state, id, BPF_REF_INVALID)); for (i = 0; i <= vstate->curframe; i++) __mark_ptr_or_null_regs(vstate->frame[i], id, is_null); From patchwork Sun May 17 19:57:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 219083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59856C433E0 for ; Sun, 17 May 2020 19:57:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2892C20758 for ; Sun, 17 May 2020 19:57:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="lMR3XBCo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726707AbgEQT5w (ORCPT ); Sun, 17 May 2020 15:57:52 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:3418 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726656AbgEQT5v (ORCPT ); Sun, 17 May 2020 15:57:51 -0400 Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 04HJrwjF029448 for ; Sun, 17 May 2020 12:57:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=63Ar7aU39zy5gm+BC6P7OZLvJhUvPsb/RjKG4/qHPVQ=; b=lMR3XBCoEzrtJ5+l6mPt2kXN2dfW2QTHRDZ9VlC0slCVmieJncZwtSfm2zLKSvweO6qJ Uww0xmMVRSpD+XRZI03/68clnaNGWXkvOjGIoD3B8TvgyVfJYTR7GcskxgJ/3ZIUhaVX B330O7JvIENo0slXyTJtzYcTqH/cyYDKFMA= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 312dpwtxg6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sun, 17 May 2020 12:57:50 -0700 Received: from intmgw001.03.ash8.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1847.3; Sun, 17 May 2020 12:57:48 -0700 Received: by devbig012.ftw2.facebook.com (Postfix, from userid 137359) id E3A5B2EC32DC; Sun, 17 May 2020 12:57:43 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Andrii Nakryiko Smtp-Origin-Hostname: devbig012.ftw2.facebook.com To: , , , CC: , , Andrii Nakryiko , "Paul E . McKenney" , Jonathan Lemon Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v2 bpf-next 4/7] libbpf: add BPF ring buffer support Date: Sun, 17 May 2020 12:57:24 -0700 Message-ID: <20200517195727.279322-5-andriin@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200517195727.279322-1-andriin@fb.com> References: <20200517195727.279322-1-andriin@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-17_08:2020-05-15,2020-05-17 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 mlxscore=0 phishscore=0 malwarescore=0 clxscore=1015 impostorscore=0 spamscore=0 mlxlogscore=919 lowpriorityscore=0 suspectscore=25 bulkscore=0 priorityscore=1501 cotscore=-2147483648 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2005170182 X-FB-Internal: deliver Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Declaring and instantiating BPF ring buffer doesn't require any changes to libbpf, as it's just another type of maps. So using existing BTF-defined maps syntax with __uint(type, BPF_MAP_TYPE_RINGBUF) and __uint(max_elements, ) is all that's necessary to create and use BPF ring buffer. This patch adds BPF ring buffer consumer to libbpf. It is very similar to perf_buffer implementation in terms of API, but also attempts to fix some minor problems and inconveniences with existing perf_buffer API. ring_buffer support both single ring buffer use case (with just using ring_buffer__new()), as well as allows to add more ring buffers, each with its own callback and context. This allows to efficiently poll and consume multiple, potentially completely independent, ring buffers, using single epoll instance. The latter is actually a problem in practice for applications that are using multiple sets of perf buffers. They have to create multiple instances for struct perf_buffer and poll them independently or in a loop, each approach having its own problems (e.g., inability to use a common poll timeout). struct ring_buffer eliminates this problem by aggregating many independent ring buffer instances under the single "ring buffer manager". Second, perf_buffer's callback can't return error, so applications that need to stop polling due to error in data or data signalling the end, have to use extra mechanisms to signal that polling has to stop. ring_buffer's callback can return error, which will be passed through back to user code and can be acted upon appropariately. Two APIs allow to consume ring buffer data: - ring_buffer__poll(), which will wait for data availability notification and will consume data only from reported ring buffer(s); this API allows to efficiently use resources by reading data only when it becomes available; - ring_buffer__consume(), will attempt to read new records regardless of data availablity notification sub-system. This API is useful for cases when lowest latency is required, in expense of burning CPU resources. Signed-off-by: Andrii Nakryiko --- tools/lib/bpf/Build | 2 +- tools/lib/bpf/libbpf.h | 21 +++ tools/lib/bpf/libbpf.map | 5 + tools/lib/bpf/libbpf_probes.c | 5 + tools/lib/bpf/ringbuf.c | 285 ++++++++++++++++++++++++++++++++++ 5 files changed, 317 insertions(+), 1 deletion(-) create mode 100644 tools/lib/bpf/ringbuf.c diff --git a/tools/lib/bpf/Build b/tools/lib/bpf/Build index e3962cfbc9a6..190366d05588 100644 --- a/tools/lib/bpf/Build +++ b/tools/lib/bpf/Build @@ -1,3 +1,3 @@ libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \ netlink.o bpf_prog_linfo.o libbpf_probes.o xsk.o hashmap.o \ - btf_dump.o + btf_dump.o ringbuf.o diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index 8ea69558f0a8..e0ee0906202b 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -478,6 +478,27 @@ LIBBPF_API int bpf_get_link_xdp_id(int ifindex, __u32 *prog_id, __u32 flags); LIBBPF_API int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info, size_t info_size, __u32 flags); +/* Ring buffer APIs */ +struct ring_buffer; + +typedef int (*ring_buffer_sample_fn)(void *ctx, void *data, size_t size); + +struct ring_buffer_opts { + size_t sz; /* size of this struct, for forward/backward compatiblity */ +}; + +#define ring_buffer_opts__last_field sz + +LIBBPF_API struct ring_buffer * +ring_buffer__new(int map_fd, ring_buffer_sample_fn sample_cb, void *ctx, + const struct ring_buffer_opts *opts); +LIBBPF_API void ring_buffer__free(struct ring_buffer *rb); +LIBBPF_API int ring_buffer__add(struct ring_buffer *rb, int map_fd, + ring_buffer_sample_fn sample_cb, void *ctx); +LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); +LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb); + +/* Perf buffer APIs */ struct perf_buffer; typedef void (*perf_buffer_sample_fn)(void *ctx, int cpu, diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 0133d469d30b..51e58a6215de 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -262,4 +262,9 @@ LIBBPF_0.0.9 { bpf_link_get_fd_by_id; bpf_link_get_next_id; bpf_program__attach_iter; + ring_buffer__add; + ring_buffer__consume; + ring_buffer__free; + ring_buffer__new; + ring_buffer__poll; } LIBBPF_0.0.8; diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index 2c92059c0c90..10cd8d1891f5 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -238,6 +238,11 @@ bool bpf_probe_map_type(enum bpf_map_type map_type, __u32 ifindex) if (btf_fd < 0) return false; break; + case BPF_MAP_TYPE_RINGBUF: + key_size = 0; + value_size = 0; + max_entries = 4096; + break; case BPF_MAP_TYPE_UNSPEC: case BPF_MAP_TYPE_HASH: case BPF_MAP_TYPE_ARRAY: diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c new file mode 100644 index 000000000000..dcbb0b92f5ee --- /dev/null +++ b/tools/lib/bpf/ringbuf.c @@ -0,0 +1,285 @@ +// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +/* + * Ring buffer operations. + * + * Copyright (C) 2020 Facebook, Inc. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "libbpf.h" +#include "libbpf_internal.h" +#include "bpf.h" + +/* make sure libbpf doesn't use kernel-only integer typedefs */ +#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64 + +struct ring { + ring_buffer_sample_fn sample_cb; + void *ctx; + void *data; + unsigned long *consumer_pos; + unsigned long *producer_pos; + unsigned long mask; + int map_fd; +}; + +struct ring_buffer { + struct epoll_event *events; + struct ring *rings; + size_t page_size; + int epoll_fd; + int ring_cnt; +}; + +static void ringbuf_unmap_ring(struct ring_buffer *rb, struct ring *r) +{ + if (r->consumer_pos) { + munmap(r->consumer_pos, rb->page_size); + r->consumer_pos = NULL; + } + if (r->producer_pos) { + munmap(r->producer_pos, rb->page_size + 2 * (r->mask + 1)); + r->producer_pos = NULL; + } +} + +/* Add extra RINGBUF maps to this ring buffer manager */ +int ring_buffer__add(struct ring_buffer *rb, int map_fd, + ring_buffer_sample_fn sample_cb, void *ctx) +{ + struct bpf_map_info info; + __u32 len = sizeof(info); + struct epoll_event *e; + struct ring *r; + void *tmp; + int err; + + memset(&info, 0, sizeof(info)); + + err = bpf_obj_get_info_by_fd(map_fd, &info, &len); + if (err) { + err = -errno; + pr_warn("ringbuf: failed to get map info for fd=%d: %d\n", + map_fd, err); + return err; + } + + if (info.type != BPF_MAP_TYPE_RINGBUF) { + pr_warn("ringbuf: map fd=%d is not BPF_MAP_TYPE_RINGBUF\n", + map_fd); + return -EINVAL; + } + + tmp = reallocarray(rb->rings, rb->ring_cnt + 1, sizeof(*rb->rings)); + if (!tmp) + return -ENOMEM; + rb->rings = tmp; + + tmp = reallocarray(rb->events, rb->ring_cnt + 1, sizeof(*rb->events)); + if (!tmp) + return -ENOMEM; + rb->events = tmp; + + r = &rb->rings[rb->ring_cnt]; + memset(r, 0, sizeof(*r)); + + r->map_fd = map_fd; + r->sample_cb = sample_cb; + r->ctx = ctx; + r->mask = info.max_entries - 1; + + /* Map writable consumer page */ + tmp = mmap(NULL, rb->page_size, PROT_READ | PROT_WRITE, MAP_SHARED, + map_fd, 0); + if (tmp == MAP_FAILED) { + err = -errno; + pr_warn("ringbuf: failed to mmap consumer page for map fd=%d: %d\n", + map_fd, err); + return err; + } + r->consumer_pos = tmp; + + /* Map read-only producer page and data pages. We map twice as big + * data size to allow simple reading of samples that wrap around the + * end of a ring buffer. See kernel implementation for details. + * */ + tmp = mmap(NULL, rb->page_size + 2 * info.max_entries, PROT_READ, + MAP_SHARED, map_fd, rb->page_size); + if (tmp == MAP_FAILED) { + err = -errno; + ringbuf_unmap_ring(rb, r); + pr_warn("ringbuf: failed to mmap data pages for map fd=%d: %d\n", + map_fd, err); + return err; + } + r->producer_pos = tmp; + r->data = tmp + rb->page_size; + + e = &rb->events[rb->ring_cnt]; + memset(e, 0, sizeof(*e)); + + e->events = EPOLLIN; + e->data.fd = rb->ring_cnt; + if (epoll_ctl(rb->epoll_fd, EPOLL_CTL_ADD, map_fd, e) < 0) { + err = -errno; + ringbuf_unmap_ring(rb, r); + pr_warn("ringbuf: failed to epoll add map fd=%d: %d\n", + map_fd, err); + return err; + } + + rb->ring_cnt++; + return 0; +} + +void ring_buffer__free(struct ring_buffer *rb) +{ + int i; + + if (!rb) + return; + + for (i = 0; i < rb->ring_cnt; ++i) + ringbuf_unmap_ring(rb, &rb->rings[i]); + if (rb->epoll_fd >= 0) + close(rb->epoll_fd); + + free(rb->events); + free(rb->rings); + free(rb); +} + +struct ring_buffer * +ring_buffer__new(int map_fd, ring_buffer_sample_fn sample_cb, void *ctx, + const struct ring_buffer_opts *opts) +{ + struct ring_buffer *rb; + int err; + + if (!OPTS_VALID(opts, ring_buffer_opts)) + return NULL; + + rb = calloc(1, sizeof(*rb)); + if (!rb) + return NULL; + + rb->page_size = getpagesize(); + + rb->epoll_fd = epoll_create1(EPOLL_CLOEXEC); + if (rb->epoll_fd < 0) { + err = -errno; + pr_warn("ringbuf: failed to create epoll instance: %d\n", err); + goto err_out; + } + + err = ring_buffer__add(rb, map_fd, sample_cb, ctx); + if (err) + goto err_out; + + return rb; + +err_out: + ring_buffer__free(rb); + return NULL; +} + +static inline int roundup_len(__u32 len) +{ + /* clear out top 2 bits */ + len <<= 2; + len >>= 2; + /* add length prefix */ + len += BPF_RINGBUF_HDR_SZ; + /* round up to 8 byte alignment */ + return (len + 7) / 8 * 8; +} + +static int ringbuf_process_ring(struct ring* r) +{ + int *len_ptr, len, err, cnt = 0; + unsigned long cons_pos, prod_pos; + bool got_new_data; + void *sample; + + cons_pos = smp_load_acquire(r->consumer_pos); + do { + got_new_data = false; + prod_pos = smp_load_acquire(r->producer_pos); + while (cons_pos < prod_pos) { + len_ptr = r->data + (cons_pos & r->mask); + len = smp_load_acquire(len_ptr); + + /* sample not committed yet, bail out for now */ + if (len & BPF_RINGBUF_BUSY_BIT) + goto done; + + got_new_data = true; + cons_pos += roundup_len(len); + + if ((len & BPF_RINGBUF_DISCARD_BIT) == 0) { + sample = (void *)len_ptr + BPF_RINGBUF_HDR_SZ; + err = r->sample_cb(r->ctx, sample, len); + if (err) { + /* update consumer pos and bail out */ + smp_store_release(r->consumer_pos, + cons_pos); + return err; + } + cnt++; + } + + smp_store_release(r->consumer_pos, cons_pos); + } + } while (got_new_data); +done: + return cnt; +} + +/* Consume available ring buffer(s) data without event polling. + * Returns number of records consumed across all registered ring buffers, or + * negative number if any of the callbacks return error. + */ +int ring_buffer__consume(struct ring_buffer *rb) +{ + int i, err, res = 0; + + for (i = 0; i < rb->ring_cnt; i++) { + struct ring *ring = &rb->rings[i]; + + err = ringbuf_process_ring(ring); + if (err < 0) + return err; + res += err; + } + return res; +} + +/* Poll for available data and consume records, if any are available. + * Returns number of records consumed, or negative number, if any of the + * registered callbacks returned error. + */ +int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms) +{ + int i, cnt, err, res = 0; + + cnt = epoll_wait(rb->epoll_fd, rb->events, rb->ring_cnt, timeout_ms); + for (i = 0; i < cnt; i++) { + __u32 ring_id = rb->events[i].data.fd; + struct ring *ring = &rb->rings[ring_id]; + + err = ringbuf_process_ring(ring); + if (err < 0) + return err; + res += cnt; + } + return cnt < 0 ? -errno : res; +} From patchwork Sun May 17 19:57:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 219081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5ECEC433E1 for ; Sun, 17 May 2020 19:58:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87A7C20758 for ; Sun, 17 May 2020 19:58:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="ljuVYDJK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726855AbgEQT6F (ORCPT ); Sun, 17 May 2020 15:58:05 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:37624 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726715AbgEQT5y (ORCPT ); Sun, 17 May 2020 15:57:54 -0400 Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 04HJqoGj023455 for ; Sun, 17 May 2020 12:57:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=In1dYdsF3WgnndXMLDTL1E4M0iOAz+O9fxDc5Uy4y1s=; b=ljuVYDJKqa5vrkUpRWwT5L6QJCSyVmsvG9OtnSKy+S1yiSeoCJ7Dv48QodJ6TcvER1yw 7RCkSkaQJgjr9Bq1fq7TKcHPG5YDsygmVBxPeuelcP3OgGpQ3SaFGyuMU7zipj18vBWr GTuzFr5McskZYqv9LZ2kr0KGUWHZifD31A4= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 312dwq2nkd-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sun, 17 May 2020 12:57:52 -0700 Received: from intmgw001.03.ash8.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1847.3; Sun, 17 May 2020 12:57:48 -0700 Received: by devbig012.ftw2.facebook.com (Postfix, from userid 137359) id 1DD7C2EC32DC; Sun, 17 May 2020 12:57:46 -0700 (PDT) Smtp-Origin-Hostprefix: devbig From: Andrii Nakryiko Smtp-Origin-Hostname: devbig012.ftw2.facebook.com To: , , , CC: , , Andrii Nakryiko , "Paul E . McKenney" , Jonathan Lemon Smtp-Origin-Cluster: ftw2c04 Subject: [PATCH v2 bpf-next 5/7] selftests/bpf: add BPF ringbuf selftests Date: Sun, 17 May 2020 12:57:25 -0700 Message-ID: <20200517195727.279322-6-andriin@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200517195727.279322-1-andriin@fb.com> References: <20200517195727.279322-1-andriin@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-17_08:2020-05-15,2020-05-17 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 suspectscore=25 spamscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 priorityscore=1501 clxscore=1015 phishscore=0 cotscore=-2147483648 lowpriorityscore=0 adultscore=0 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2005170182 X-FB-Internal: deliver Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Both singleton BPF ringbuf and BPF ringbuf with map-in-map use cases are tested. Also reserve+submit/discards and output variants of API are validated. Signed-off-by: Andrii Nakryiko --- .../selftests/bpf/prog_tests/ringbuf.c | 211 ++++++++++++++++++ .../selftests/bpf/prog_tests/ringbuf_multi.c | 102 +++++++++ .../selftests/bpf/progs/test_ringbuf.c | 78 +++++++ .../selftests/bpf/progs/test_ringbuf_multi.c | 77 +++++++ 4 files changed, 468 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/ringbuf.c create mode 100644 tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c create mode 100644 tools/testing/selftests/bpf/progs/test_ringbuf.c create mode 100644 tools/testing/selftests/bpf/progs/test_ringbuf_multi.c diff --git a/tools/testing/selftests/bpf/prog_tests/ringbuf.c b/tools/testing/selftests/bpf/prog_tests/ringbuf.c new file mode 100644 index 000000000000..bb8541f240e2 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/ringbuf.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "test_ringbuf.skel.h" + +#define EDONE 7777 + +static int duration = 0; + +struct sample { + int pid; + int seq; + long value; + char comm[16]; +}; + +static int sample_cnt; + +static int process_sample(void *ctx, void *data, size_t len) +{ + struct sample *s = data; + + sample_cnt++; + + switch (s->seq) { + case 0: + CHECK(s->value != 333, "sample1_value", "exp %ld, got %ld\n", + 333L, s->value); + return 0; + case 1: + CHECK(s->value != 777, "sample2_value", "exp %ld, got %ld\n", + 777L, s->value); + return -EDONE; + default: + /* we don't care about the rest */ + return 0; + } +} + +static struct test_ringbuf *skel; +static struct ring_buffer *ringbuf; + +static void trigger_samples() +{ + skel->bss->dropped = 0; + skel->bss->total = 0; + skel->bss->discarded = 0; + + /* trigger exactly two samples */ + skel->bss->value = 333; + syscall(__NR_getpgid); + skel->bss->value = 777; + syscall(__NR_getpgid); +} + +static void *poll_thread(void *input) +{ + long timeout = (long)input; + + return (void *)(long)ring_buffer__poll(ringbuf, timeout); +} + +void test_ringbuf(void) +{ + const size_t rec_sz = BPF_RINGBUF_HDR_SZ + sizeof(struct sample); + pthread_t thread; + long bg_ret = -1; + int err; + + skel = test_ringbuf__open_and_load(); + if (CHECK(!skel, "skel_open_load", "skeleton open&load failed\n")) + return; + + /* only trigger BPF program for current process */ + skel->bss->pid = getpid(); + + ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf), + process_sample, NULL, NULL); + if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n")) + goto cleanup; + + err = test_ringbuf__attach(skel); + if (CHECK(err, "skel_attach", "skeleton attachment failed: %d\n", err)) + goto cleanup; + + trigger_samples(); + + /* 2 submitted + 1 discarded records */ + CHECK(skel->bss->avail_data != 3 * rec_sz, + "err_avail_size", "exp %ld, got %ld\n", + 3L * rec_sz, skel->bss->avail_data); + CHECK(skel->bss->ring_size != 4096, + "err_ring_size", "exp %ld, got %ld\n", + 4096L, skel->bss->ring_size); + CHECK(skel->bss->cons_pos != 0, + "err_cons_pos", "exp %ld, got %ld\n", + 0L, skel->bss->cons_pos); + CHECK(skel->bss->prod_pos != 3 * rec_sz, + "err_prod_pos", "exp %ld, got %ld\n", + 3L * rec_sz, skel->bss->prod_pos); + + /* poll for samples */ + err = ring_buffer__poll(ringbuf, -1); + + /* -EDONE is used as an indicator that we are done */ + if (CHECK(err != -EDONE, "err_done", "done err: %d\n", err)) + goto cleanup; + + /* we expect extra polling to return nothing */ + err = ring_buffer__poll(ringbuf, 0); + if (CHECK(err != 0, "extra_samples", "poll result: %d\n", err)) + goto cleanup; + + CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n", + 0L, skel->bss->dropped); + CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n", + 2L, skel->bss->total); + CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n", + 1L, skel->bss->discarded); + + /* now validate consumer position is updated and returned */ + trigger_samples(); + CHECK(skel->bss->cons_pos != 3 * rec_sz, + "err_cons_pos", "exp %ld, got %ld\n", + 3L * rec_sz, skel->bss->cons_pos); + err = ring_buffer__poll(ringbuf, -1); + CHECK(err <= 0, "poll_err", "err %d\n", err); + + /* start poll in background w/ long timeout */ + err = pthread_create(&thread, NULL, poll_thread, (void *)(long)10000); + if (CHECK(err, "bg_poll", "pthread_create failed: %d\n", err)) + goto cleanup; + + /* turn off notifications now */ + skel->bss->flags = BPF_RB_NO_WAKEUP; + + /* give background thread a bit of a time */ + usleep(50000); + trigger_samples(); + /* sleeping arbitrarily is bad, but no better way to know that + * epoll_wait() **DID NOT** unblock in background thread + */ + usleep(50000); + /* background poll should still be blocked */ + err = pthread_tryjoin_np(thread, (void **)&bg_ret); + if (CHECK(err != EBUSY, "try_join", "err %d\n", err)) + goto cleanup; + + /* BPF side did everything right */ + CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n", + 0L, skel->bss->dropped); + CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n", + 2L, skel->bss->total); + CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n", + 1L, skel->bss->discarded); + + /* clear flags to return to "adaptive" notification mode */ + skel->bss->flags = 0; + + /* produce new samples, no notification should be triggered, because + * consumer is now behind + */ + trigger_samples(); + + /* background poll should still be blocked */ + err = pthread_tryjoin_np(thread, (void **)&bg_ret); + if (CHECK(err != EBUSY, "try_join", "err %d\n", err)) + goto cleanup; + + /* now force notifications */ + skel->bss->flags = BPF_RB_FORCE_WAKEUP; + sample_cnt = 0; + trigger_samples(); + + /* now we should get a pending notification */ + usleep(50000); + err = pthread_tryjoin_np(thread, (void **)&bg_ret); + if (CHECK(err, "join_bg", "err %d\n", err)) + goto cleanup; + + if (CHECK(bg_ret != 1, "bg_ret", "epoll_wait result: %ld", bg_ret)) + goto cleanup; + + /* 3 rounds, 2 samples each */ + CHECK(sample_cnt != 6, "wrong_sample_cnt", + "expected to see %d samples, got %d\n", 6, sample_cnt); + + /* BPF side did everything right */ + CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n", + 0L, skel->bss->dropped); + CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n", + 2L, skel->bss->total); + CHECK(skel->bss->discarded != 1, "err_discarded", "exp %ld, got %ld\n", + 1L, skel->bss->discarded); + + test_ringbuf__detach(skel); +cleanup: + ring_buffer__free(ringbuf); + test_ringbuf__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c b/tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c new file mode 100644 index 000000000000..78e450609803 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/ringbuf_multi.c @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include +#include +#include "test_ringbuf_multi.skel.h" + +static int duration = 0; + +struct sample { + int pid; + int seq; + long value; + char comm[16]; +}; + +static int process_sample(void *ctx, void *data, size_t len) +{ + int ring = (unsigned long)ctx; + struct sample *s = data; + + switch (s->seq) { + case 0: + CHECK(ring != 1, "sample1_ring", "exp %d, got %d\n", 1, ring); + CHECK(s->value != 333, "sample1_value", "exp %ld, got %ld\n", + 333L, s->value); + break; + case 1: + CHECK(ring != 2, "sample2_ring", "exp %d, got %d\n", 2, ring); + CHECK(s->value != 777, "sample2_value", "exp %ld, got %ld\n", + 777L, s->value); + break; + default: + CHECK(true, "extra_sample", "unexpected sample seq %d, val %ld\n", + s->seq, s->value); + return -1; + } + + return 0; +} + +void test_ringbuf_multi(void) +{ + struct test_ringbuf_multi *skel; + struct ring_buffer *ringbuf; + int err; + + skel = test_ringbuf_multi__open_and_load(); + if (CHECK(!skel, "skel_open_load", "skeleton open&load failed\n")) + return; + + /* only trigger BPF program for current process */ + skel->bss->pid = getpid(); + + ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf1), + process_sample, (void *)(long)1, NULL); + if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n")) + goto cleanup; + + err = ring_buffer__add(ringbuf, bpf_map__fd(skel->maps.ringbuf2), + process_sample, (void *)(long)2); + if (CHECK(err, "ringbuf_add", "failed to add another ring\n")) + goto cleanup; + + err = test_ringbuf_multi__attach(skel); + if (CHECK(err, "skel_attach", "skeleton attachment failed: %d\n", err)) + goto cleanup; + + /* trigger few samples, some will be skipped */ + skel->bss->target_ring = 0; + skel->bss->value = 333; + syscall(__NR_getpgid); + + /* skipped, no ringbuf in slot 1 */ + skel->bss->target_ring = 1; + skel->bss->value = 555; + syscall(__NR_getpgid); + + skel->bss->target_ring = 2; + skel->bss->value = 777; + syscall(__NR_getpgid); + + /* poll for samples, should get 2 ringbufs back */ + err = ring_buffer__poll(ringbuf, -1); + if (CHECK(err != 4, "poll_res", "expected 4 records, got %d\n", err)) + goto cleanup; + + /* expect extra polling to return nothing */ + err = ring_buffer__poll(ringbuf, 0); + if (CHECK(err < 0, "extra_samples", "poll result: %d\n", err)) + goto cleanup; + + CHECK(skel->bss->dropped != 0, "err_dropped", "exp %ld, got %ld\n", + 0L, skel->bss->dropped); + CHECK(skel->bss->skipped != 1, "err_skipped", "exp %ld, got %ld\n", + 1L, skel->bss->skipped); + CHECK(skel->bss->total != 2, "err_total", "exp %ld, got %ld\n", + 2L, skel->bss->total); + +cleanup: + ring_buffer__free(ringbuf); + test_ringbuf_multi__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/test_ringbuf.c b/tools/testing/selftests/bpf/progs/test_ringbuf.c new file mode 100644 index 000000000000..79114eafd283 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_ringbuf.c @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook + +#include +#include + +char _license[] SEC("license") = "GPL"; + +struct sample { + int pid; + int seq; + long value; + char comm[16]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 1 << 12); +} ringbuf SEC(".maps"); + +/* inputs */ +int pid = 0; +long value = 0; +long flags = 0; + +/* outputs */ +long total = 0; +long discarded = 0; +long dropped = 0; + +long avail_data = 0; +long ring_size = 0; +long cons_pos = 0; +long prod_pos = 0; + +/* inner state */ +long seq = 0; + +SEC("tp/syscalls/sys_enter_getpgid") +int test_ringbuf(void *ctx) +{ + int cur_pid = bpf_get_current_pid_tgid() >> 32; + struct sample *sample; + int zero = 0; + + if (cur_pid != pid) + return 0; + + sample = bpf_ringbuf_reserve(&ringbuf, sizeof(*sample), 0); + if (!sample) { + __sync_fetch_and_add(&dropped, 1); + return 1; + } + + sample->pid = pid; + bpf_get_current_comm(sample->comm, sizeof(sample->comm)); + sample->value = value; + + sample->seq = seq++; + __sync_fetch_and_add(&total, 1); + + if (sample->seq & 1) { + /* copy from reserved sample to a new one... */ + bpf_ringbuf_output(&ringbuf, sample, sizeof(*sample), flags); + /* ...and then discard reserved sample */ + bpf_ringbuf_discard(sample, flags); + __sync_fetch_and_add(&discarded, 1); + } else { + bpf_ringbuf_submit(sample, flags); + } + + avail_data = bpf_ringbuf_query(&ringbuf, BPF_RB_AVAIL_DATA); + ring_size = bpf_ringbuf_query(&ringbuf, BPF_RB_RING_SIZE); + cons_pos = bpf_ringbuf_query(&ringbuf, BPF_RB_CONS_POS); + prod_pos = bpf_ringbuf_query(&ringbuf, BPF_RB_PROD_POS); + + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/test_ringbuf_multi.c b/tools/testing/selftests/bpf/progs/test_ringbuf_multi.c new file mode 100644 index 000000000000..7eb85dd9cd66 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_ringbuf_multi.c @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook + +#include +#include + +char _license[] SEC("license") = "GPL"; + +struct sample { + int pid; + int seq; + long value; + char comm[16]; +}; + +struct ringbuf_map { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 1 << 12); +} ringbuf1 SEC(".maps"), + ringbuf2 SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS); + __uint(max_entries, 4); + __type(key, int); + __array(values, struct ringbuf_map); +} ringbuf_arr SEC(".maps") = { + .values = { + [0] = &ringbuf1, + [2] = &ringbuf2, + }, +}; + +/* inputs */ +int pid = 0; +int target_ring = 0; +long value = 0; + +/* outputs */ +long total = 0; +long dropped = 0; +long skipped = 0; + +SEC("tp/syscalls/sys_enter_getpgid") +int test_ringbuf(void *ctx) +{ + int cur_pid = bpf_get_current_pid_tgid() >> 32; + struct sample *sample; + void *rb; + int zero = 0; + + if (cur_pid != pid) + return 0; + + rb = bpf_map_lookup_elem(&ringbuf_arr, &target_ring); + if (!rb) { + skipped += 1; + return 1; + } + + sample = bpf_ringbuf_reserve(rb, sizeof(*sample), 0); + if (!sample) { + dropped += 1; + return 1; + } + + sample->pid = pid; + bpf_get_current_comm(sample->comm, sizeof(sample->comm)); + sample->value = value; + + sample->seq = total; + total += 1; + + bpf_ringbuf_submit(sample, 0); + + return 0; +}