From patchwork Wed Aug 28 14:32:23 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neil Williams X-Patchwork-Id: 19563 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-vc0-f200.google.com (mail-vc0-f200.google.com [209.85.220.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 0D09525E58 for ; Wed, 28 Aug 2013 14:32:26 +0000 (UTC) Received: by mail-vc0-f200.google.com with SMTP id hf12sf6758310vcb.3 for ; Wed, 28 Aug 2013 07:32:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:delivered-to:mime-version:to:from:subject :message-id:date:reply-to:sender:errors-to:precedence :x-original-sender:x-original-authentication-results:mailing-list :list-id:list-post:list-help:list-archive:list-unsubscribe :content-type; bh=GKmLPYb/kww2f+YGHDNzzwtfrWeSzEikzZ68Bvk0imQ=; b=MUF17T+MZB7f5t7GMLJmzYjuFZBT1vzFUoLbpKtJGXJGote++CMpkhqYJqe782M0ku apw7Bb4Y5w1/C8endjlZD3lXxv2WJN+34ukHZ9FAt5qVNsJF1IkIn60QRHQDC6saT3pv E93z74ovjJ2gt5Z2MpWtuzvuImqUHaFlQhLG2wLUaqqYCd5n6UvfHhh0SE+dEw3GRKiy DSbrl/gg/Qv82gS+22LXKFFWZFMEAZtq+OPPd84ZME/7jwQa4WZVqfu1hiMXeWcaKU02 9SjHma51hDWDc8R4lEpHYIDlm+ZD6OqxBEgf4tsxWdZxU2FQIWrYghiKKp/wXcV3qEkK hlHg== X-Received: by 10.236.124.33 with SMTP id w21mr9883015yhh.15.1377700345433; Wed, 28 Aug 2013 07:32:25 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.82.114 with SMTP id h18ls404753qey.12.gmail; Wed, 28 Aug 2013 07:32:25 -0700 (PDT) X-Received: by 10.58.155.68 with SMTP id vu4mr13791529veb.21.1377700345205; Wed, 28 Aug 2013 07:32:25 -0700 (PDT) Received: from mail-vc0-f173.google.com (mail-vc0-f173.google.com [209.85.220.173]) by mx.google.com with ESMTPS id fx15si6601149vec.72.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 28 Aug 2013 07:32:25 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.173 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.173; Received: by mail-vc0-f173.google.com with SMTP id id13so4058477vcb.18 for ; Wed, 28 Aug 2013 07:32:25 -0700 (PDT) X-Gm-Message-State: ALoCoQkhFJ7MKOf37ACea9mHzjY3xqEgvKWmJDP1zL2oJSPbso8dqqKWA/NLK21DvGyTvmqxODL8 X-Received: by 10.52.22.14 with SMTP id z14mr73035vde.49.1377700344944; Wed, 28 Aug 2013 07:32:24 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.174.196 with SMTP id u4csp359098vcz; Wed, 28 Aug 2013 07:32:24 -0700 (PDT) X-Received: by 10.194.119.166 with SMTP id kv6mr744189wjb.89.1377700343704; Wed, 28 Aug 2013 07:32:23 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id op2si9499051wjc.59.1969.12.31.16.00.00 (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 28 Aug 2013 07:32:23 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1VEgnH-0001Wk-2b for ; Wed, 28 Aug 2013 14:32:23 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 05984E028C for ; Wed, 28 Aug 2013 14:32:23 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dashboard X-Launchpad-Branch: ~linaro-validation/lava-dashboard/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 415 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dashboard/trunk] Rev 415: Landing MultiNode - add support in the dashboard. Message-Id: <20130828143223.27465.15435.launchpad@ackee.canonical.com> Date: Wed, 28 Aug 2013 14:32:23 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: list X-Generated-By: Launchpad (canonical.com); Revision="16738"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 181eb5e9652f949c90884f375270fffe58f1fd41 X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: noreply@launchpad.net X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.173 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Merge authors: Fu Wei (fu-wei) Neil Williams (codehelp) Related merge proposals: https://code.launchpad.net/~linaro-automation/lava-dashboard/multinode/+merge/181101 proposed by: Neil Williams (codehelp) review: Approve - Neil Williams (codehelp) ------------------------------------------------------------ revno: 415 [merge] committer: Neil Williams branch nick: lava-dashboard timestamp: Wed 2013-08-28 15:28:04 +0100 message: Landing MultiNode - add support in the dashboard. Neil Williams 2013-08-24 [merge] Neil Williams 2013-08-23 Add support for checking authentication of the pending bundles by porting the stream check code from _put without allowing the pending bundle into the database before aggregation. Neil Williams 2013-08-15 [merge] Fu Wei 2013-08-14 Add "Device" row in bundle "Test Runs" Fu Wei 2013-08-14 Add "Device" row in bundle "Test Runs" Neil Williams 2013-08-15 [merge] Neil Williams 2013-08-14 Change to storing the pending bundles in a file and explicitly check for exceptions which would cause a failure to submit the aggregated result bundle. Neil Williams 2013-08-05 MultiNode support to put_pending result bundles for jobs with a sub_id ending with .0. The MultiNode job with sub_id ending with .0 will be required by LAVA Coordinator to wait until all other results for that group have been posted as pending. Once the wait is cleared, the job aggregates its own results with the group pending results as a single bundle using put_group. modified: dashboard_app/templates/dashboard_app/_test_run_list_table.html dashboard_app/xmlrpc.py --- lp:lava-dashboard https://code.launchpad.net/~linaro-validation/lava-dashboard/trunk You are subscribed to branch lp:lava-dashboard. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dashboard/trunk/+edit-subscription === modified file 'dashboard_app/templates/dashboard_app/_test_run_list_table.html' --- dashboard_app/templates/dashboard_app/_test_run_list_table.html 2013-02-25 09:25:30 +0000 +++ dashboard_app/templates/dashboard_app/_test_run_list_table.html 2013-08-14 11:32:27 +0000 @@ -2,6 +2,7 @@ + @@ -13,6 +14,11 @@ {% for test_run in test_run_list %} + {% for attribute in test_run.attributes.all %} + {% if attribute.name == "target" %} + + {% endif %} + {% endfor %} === modified file 'dashboard_app/xmlrpc.py' --- dashboard_app/xmlrpc.py 2013-05-02 10:35:29 +0000 +++ dashboard_app/xmlrpc.py 2013-08-23 10:31:07 +0000 @@ -2,7 +2,7 @@ # # Author: Zygmunt Krynicki # -# This file is part of Launch Control. +# This file is part of LAVA Dashboard # # Launch Control is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License version 3 @@ -25,7 +25,9 @@ import logging import re import xmlrpclib - +import hashlib +import json +import os from django.contrib.auth.models import User, Group from django.core.urlresolvers import reverse from django.db import IntegrityError, DatabaseError @@ -105,9 +107,9 @@ logging.debug("Getting bundle stream") bundle_stream = BundleStream.objects.accessible_by_principal(self.user).get(pathname=pathname) except BundleStream.DoesNotExist: - logging.debug("Bundle stream does not exists, aborting") + logging.debug("Bundle stream does not exist, aborting") raise xmlrpclib.Fault(errors.NOT_FOUND, - "Bundle stream not found") + "Bundle stream not found") if not bundle_stream.can_upload(self.user): raise xmlrpclib.Fault( errors.FORBIDDEN, "You cannot upload to this stream") @@ -243,6 +245,186 @@ 'dashboard_app.views.redirect_to_bundle', kwargs={'content_sha1':bundle.content_sha1})) + def put_pending(self, content, pathname, group_name): + """ + Name + ---- + `put_pending` (`content`, `pathname`, `group_name`) + + Description + ----------- + MultiNode internal call. + + Stores the bundle until the coordinator allows the complete + bundle list to be aggregated from the list and submitted by put_group + + Arguments + --------- + `content`: string + Full text of the bundle. This *MUST* be a valid JSON + document and it *SHOULD* match the "Dashboard Bundle Format + 1.0" schema. The SHA1 of the content *MUST* be unique or a + ``Fault(409, "...")`` is raised. This is used to protect + from simple duplicate submissions. + `pathname`: string + Pathname of the bundle stream where a new bundle should + be created and stored. This argument *MUST* designate a + pre-existing bundle stream or a ``Fault(404, "...")`` exception + is raised. In addition the user *MUST* have access + permission to upload bundles there or a ``Fault(403, "...")`` + exception is raised. See below for access rules. + `group_name`: string + Unique ID of the MultiNode group. Other pending bundles will + be aggregated into a single result bundle for this group. + + Return value + ------------ + If all goes well this function returns the SHA1 of the content. + + Exceptions raised + ----------------- + 404 + Either: + + - Bundle stream not found + - Uploading to specified stream is not permitted + 409 + Duplicate bundle content + + Rules for bundle stream access + ------------------------------ + The following rules govern bundle stream upload access rights: + - all anonymous streams are accessible + - personal streams are accessible to owners + - team streams are accessible to team members + + """ + try: + logging.debug("Getting bundle stream") + bundle_stream = BundleStream.objects.accessible_by_principal(self.user).get(pathname=pathname) + except BundleStream.DoesNotExist: + logging.debug("Bundle stream does not exist, aborting") + raise xmlrpclib.Fault(errors.NOT_FOUND, + "Bundle stream not found") + if not bundle_stream.can_upload(self.user): + raise xmlrpclib.Fault( + errors.FORBIDDEN, "You cannot upload to this stream") + try: + # add this to a list which put_group can use. + sha1 = hashlib.sha1() + sha1.update(content) + hexdigest = sha1.hexdigest() + groupfile = "/tmp/%s" % group_name + with open(groupfile, "a+") as grp_file: + grp_file.write("%s\n" % content) + return hexdigest + except Exception as e: + logging.debug("Dashboard pending submission caused an exception: %s" % e) + + def put_group(self, content, content_filename, pathname, group_name): + """ + Name + ---- + `put_group` (`content`, `content_filename`, `pathname`, `group_name`) + + Description + ----------- + MultiNode internal call. + + Adds the final bundle to the list, aggregates the list + into a single group bundle and submits the group bundle. + + Arguments + --------- + `content`: string + Full text of the bundle. This *MUST* be a valid JSON + document and it *SHOULD* match the "Dashboard Bundle Format + 1.0" schema. The SHA1 of the content *MUST* be unique or a + ``Fault(409, "...")`` is raised. This is used to protect + from simple duplicate submissions. + `content_filename`: string + Name of the file that contained the text of the bundle. The + `content_filename` can be an arbitrary string and will be + stored along with the content for reference. + `pathname`: string + Pathname of the bundle stream where a new bundle should + be created and stored. This argument *MUST* designate a + pre-existing bundle stream or a ``Fault(404, "...")`` exception + is raised. In addition the user *MUST* have access + permission to upload bundles there or a ``Fault(403, "...")`` + exception is raised. See below for access rules. + `group_name`: string + Unique ID of the MultiNode group. Other pending bundles will + be aggregated into a single result bundle for this group. At + least one other bundle must have already been submitted as + pending for the specified MultiNode group. LAVA Coordinator + causes the parent job to wait until all nodes have been marked + as having pending bundles, even if some bundles are empty. + + Return value + ------------ + If all goes well this function returns the full URL of the bundle. + + Exceptions raised + ----------------- + ValueError: + One or more bundles could not be converted to JSON prior + to aggregation. + 404 + Either: + + - Bundle stream not found + - Uploading to specified stream is not permitted + 409 + Duplicate bundle content + + Rules for bundle stream access + ------------------------------ + The following rules govern bundle stream upload access rights: + - all anonymous streams are accessible + - personal streams are accessible to owners + - team streams are accessible to team members + + """ + grp_file = "/tmp/%s" % group_name + bundle_set = {} + bundle_set[group_name] = [] + if os.path.isfile(grp_file): + with open(grp_file, "r") as grp_data: + grp_list = grp_data.readlines() + for testrun in grp_list: + bundle_set[group_name].append(json.loads(testrun)) + # Note: now that we have the data from the group, the group data file could be re-used + # as an error log which is simpler than debugging through XMLRPC. + else: + raise ValueError("Aggregation failure for %s - check coordinator rpc_delay?" % group_name) + group_tests = [] + try: + json_data = json.loads(content) + except ValueError: + logging.debug("Invalid JSON content within the sub_id zero bundle") + json_data = None + try: + bundle_set[group_name].append(json_data) + except Exception as e: + logging.debug("appending JSON caused exception %s" % e) + try: + for bundle_list in bundle_set[group_name]: + for test_run in bundle_list['test_runs']: + group_tests.append(test_run) + except Exception as e: + logging.debug("aggregating bundles caused exception %s" % e) + group_content = json.dumps({"test_runs": group_tests, "format": json_data['format']}) + bundle = self._put(group_content, content_filename, pathname) + logging.debug("Returning permalink to aggregated bundle for %s" % group_name) + permalink = self._context.request.build_absolute_uri( + reverse('dashboard_app.views.redirect_to_bundle', + kwargs={'content_sha1': bundle.content_sha1})) + # only delete the group file when things go well. + if os.path.isfile(grp_file): + os.remove(grp_file) + return permalink + def get(self, content_sha1): """ Name
{% trans "Device" %} {% trans "Test Run" %} {% trans "Test" %} {% trans "Passes" %}
{{ attribute.value }}{{ test_run.test }} results {{ test_run.test }} {{ test_run.get_summary_results.pass }}