From patchwork Tue Jul 23 12:12:27 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milo Casagrande X-Patchwork-Id: 18537 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ve0-f197.google.com (mail-ve0-f197.google.com [209.85.128.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id AE9BC25E73 for ; Tue, 23 Jul 2013 12:12:29 +0000 (UTC) Received: by mail-ve0-f197.google.com with SMTP id d10sf10690614vea.4 for ; Tue, 23 Jul 2013 05:12:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-beenthere:x-forwarded-to:x-forwarded-for:delivered-to :mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state:x-original-sender :x-original-authentication-results:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe :content-type; bh=t+ri3S8HuGMFSFJMZcXt0aDmtFSwYvve3chHX8ZV47s=; b=QdvB0tcWZ5Gfq5aOaw6vCwPr4y7jV7BD/MP6DmHPJC/RlQuEpx9nSWZyH5jGe5ZSac s/G/ZOZc1aHDPTK+jOKm4ZBf3nb7HRJ7ydMTirnJ6o1pkTQEV/hOlDfqqUz9/Subbav7 eQiTEsFEoF6Gw1lF2d2fCD7nDRte9hskJG/DZt9xg7rTFFTmJwTS7ard6wjvoeQZtFNL vaXOOnX8ibbwxoO4gj16dV03QUPM4D4c30gyPDIjLeD+Y2BmDFgQEOfzzsl+DbBhLC0a kP0En2fKGOmzqEI0IXQJqEi+WHCi0DW3tiqeubXSA6TE/40+TyV8Pt03jvbf84BiC/wd HQKA== X-Received: by 10.59.4.137 with SMTP id ce9mr3998783ved.28.1374581549237; Tue, 23 Jul 2013 05:12:29 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.97.197 with SMTP id ec5ls158355qeb.26.gmail; Tue, 23 Jul 2013 05:12:29 -0700 (PDT) X-Received: by 10.52.65.165 with SMTP id y5mr176391vds.69.1374581549035; Tue, 23 Jul 2013 05:12:29 -0700 (PDT) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx.google.com with ESMTPS id im6si8271857vdb.46.2013.07.23.05.12.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 23 Jul 2013 05:12:29 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.212.54 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.212.54; Received: by mail-vb0-f54.google.com with SMTP id q12so5508084vbe.13 for ; Tue, 23 Jul 2013 05:12:29 -0700 (PDT) X-Received: by 10.52.95.113 with SMTP id dj17mr9470524vdb.82.1374581548883; Tue, 23 Jul 2013 05:12:28 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.58.165.8 with SMTP id yu8csp103865veb; Tue, 23 Jul 2013 05:12:28 -0700 (PDT) X-Received: by 10.194.109.10 with SMTP id ho10mr22504461wjb.72.1374581547736; Tue, 23 Jul 2013 05:12:27 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id di5si13072053wjb.100.2013.07.23.05.12.27 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 23 Jul 2013 05:12:27 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1V1bS7-0003PI-4Y for ; Tue, 23 Jul 2013 12:12:27 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 1AD70E07B1 for ; Tue, 23 Jul 2013 12:12:27 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 640 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 640: Added tar-repo support in lava_test_shell. Message-Id: <20130723121227.31806.54662.launchpad@ackee.canonical.com> Date: Tue, 23 Jul 2013 12:12:27 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: list X-Generated-By: Launchpad (canonical.com); Revision="16700"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 2e3c2fe96137d4fe92848a83b355989deabc4584 X-Gm-Message-State: ALoCoQlNIcQVxV06YaDrurvacNaYCgtlpitGMcVVWCEN0pC5EAtw/lWB72apRTkrwzC6J50SMBbS X-Original-Sender: noreply@launchpad.net X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.212.54 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Merge authors: Milo Casagrande (milo) Related merge proposals: https://code.launchpad.net/~milo/lava-dispatcher/tar-repo/+merge/175274 proposed by: Milo Casagrande (milo) review: Approve - Antonio Terceiro (terceiro) ------------------------------------------------------------ revno: 640 [merge] committer: Milo Casagrande branch nick: trunk timestamp: Tue 2013-07-23 14:11:35 +0200 message: Added tar-repo support in lava_test_shell. modified: lava_dispatcher/actions/lava_test_shell.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/actions/lava_test_shell.py' --- lava_dispatcher/actions/lava_test_shell.py 2013-07-17 09:16:46 +0000 +++ lava_dispatcher/actions/lava_test_shell.py 2013-07-23 08:12:14 +0000 @@ -107,13 +107,16 @@ from datetime import datetime from glob import glob +import base64 import logging import os import pexpect import pkg_resources import shutil import stat +import StringIO import subprocess +import tarfile import tempfile import time from uuid import uuid4 @@ -187,6 +190,37 @@ logging.error('Unable to get test definition from bzr\n' + str(e)) +def _get_testdef_tar_repo(testdef_repo, tmpdir): + """Extracts the provided encoded tar archive into tmpdir.""" + tardir = os.path.join(tmpdir, 'tartestrepo') + temp_tar = os.path.join(tmpdir, "tar-repo.tar") + + try: + if not os.path.isdir(tardir): + logging.info("Creating directory to extract the tar archive into.") + os.makedirs(tardir) + + encoded_in = StringIO.StringIO(testdef_repo) + decoded_out = StringIO.StringIO() + base64.decode(encoded_in, decoded_out) + + # The following two operations can also be done in memory + # using cStringIO. + # At the moment the tar file sent is not big, but that can change. + with open(temp_tar, "w") as write_tar: + write_tar.write(decoded_out.getvalue()) + + with tarfile.open(temp_tar) as tar: + tar.extractall(path=tardir) + except (OSError, tarfile.TarError) as ex: + logging.error("Error extracting the tar archive.\n" + str(ex)) + finally: + # Remove the temporary created tar file after it has been extracted. + if os.path.isfile(temp_tar): + os.unlink(temp_tar) + return tardir + + def _get_testdef_info(testdef): metadata = {'os': '', 'devices': '', 'environment': ''} metadata['description'] = testdef['metadata'].get('description') @@ -255,8 +289,20 @@ name = testdef_repo['bzr-repo'].replace('lp:', '').split('/')[-1] info = _bzr_info(testdef_repo['bzr-repo'], repo, name) + if 'tar-repo' in testdef_repo: + repo = _get_testdef_tar_repo(testdef_repo['tar-repo'], tmpdir) + # Default info structure, since we need something, but we have + # a tar file in this case. + info = { + "project_name": "Tar archived repository", + "branch_vcs": "tar", + "branch_revision": "0", + "branch_url": repo + } + if not repo or not info: logging.debug("Unable to identify specified repository. %s" % testdef_repo) + test = testdef_repo.get('testdef', 'lavatest.yaml') with open(os.path.join(repo, test), 'r') as f: logging.info('loading test definition ...') @@ -462,13 +508,15 @@ 'items': {'type': 'object', 'properties': {'git-repo': {'type': 'string', - 'optional': True}, - 'bzr-repo': {'type': 'string', - 'optional': True}, - 'revision': {'type': 'string', - 'optional': True}, - 'testdef': {'type': 'string', - 'optional': True} + 'optional': True}, + 'bzr-repo': {'type': 'string', + 'optional': True}, + 'tar-repo': {'type': 'string', + 'optional': True}, + 'revision': {'type': 'string', + 'optional': True}, + 'testdef': {'type': 'string', + 'optional': True} }, 'additionalProperties': False}, 'optional': True