From patchwork Fri Jul 6 02:25:14 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael-Doyle Hudson X-Patchwork-Id: 9879 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 6F7D223E2F for ; Fri, 6 Jul 2012 02:25:17 +0000 (UTC) Received: from mail-gh0-f180.google.com (mail-gh0-f180.google.com [209.85.160.180]) by fiordland.canonical.com (Postfix) with ESMTP id 1FA5AA18813 for ; Fri, 6 Jul 2012 02:25:17 +0000 (UTC) Received: by ghbz12 with SMTP id z12so8773991ghb.11 for ; Thu, 05 Jul 2012 19:25:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=CsudOYsvDv1WPcd434D1FwQteoQlBJGEMvJyO9oONtE=; b=l4pzSCRcyT0yuNL9XcDgg5/50J4C8VLCCYgIkmmR0odeTA/+Fx3DY1fYGKrnNDnlEi jdmiN0g3PXgfJcyjJQcho4U5HnA+R5gRqY/Lt9j+OXaOaAyLfF/kRdWitBaY8Zzr02mh TSTQeUbETKw/epRz0sSpASOXXgX/WMeZ/IKhlIs1ornnKdR+mP2XvqrCzbOILWHP41m7 WuSjWHgenYRAczuUGZYEbYYEIptHIdJ7oIzg4jAvLHRm8oAiQ63TfSD/nXClINYN6aDM wXJfrUH+EyI4/QC4LomvSrAm+D+2lJkpxNKN89qW6TSXTllkAaTBH1v6/hX+ivssRJAR 3ZqA== Received: by 10.50.193.196 with SMTP id hq4mr1258669igc.57.1341541516186; Thu, 05 Jul 2012 19:25:16 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.24.148 with SMTP id v20csp3354ibb; Thu, 5 Jul 2012 19:25:15 -0700 (PDT) Received: by 10.180.97.135 with SMTP id ea7mr3881170wib.11.1341541514979; Thu, 05 Jul 2012 19:25:14 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id q2si4105041wiv.11.2012.07.05.19.25.14 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 05 Jul 2012 19:25:14 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1SmyEM-0000S1-8q for ; Fri, 06 Jul 2012 02:25:14 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 38DA7E00AA for ; Fri, 6 Jul 2012 02:25:14 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 345 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 345: remove most of the remaining caching code, leaving just the boot/root caching for health jobs Message-Id: <20120706022514.7327.88221.launchpad@ackee.canonical.com> Date: Fri, 06 Jul 2012 02:25:14 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15558"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 6d938238666af57d963e861072cab2470280b413 X-Gm-Message-State: ALoCoQnofTCPmKsTdRYCmiAm5sTKsdshtJy+gWNlv1fFGbMoa3f624dcyRAgLNcySAvwGjdhAaBW Merge authors: Michael Hudson-Doyle (mwhudson) Related merge proposals: https://code.launchpad.net/~mwhudson/lava-dispatcher/only-cache-health-tarballs/+merge/112680 proposed by: Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 345 [merge] committer: Michael Hudson-Doyle branch nick: trunk timestamp: Fri 2012-07-06 14:24:09 +1200 message: remove most of the remaining caching code, leaving just the boot/root caching for health jobs modified: lava_dispatcher/actions/android_deploy.py lava_dispatcher/actions/deploy.py lava_dispatcher/client/fastmodel.py lava_dispatcher/client/lmc_utils.py lava_dispatcher/client/master.py lava_dispatcher/client/qemu.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/actions/android_deploy.py' --- lava_dispatcher/actions/android_deploy.py 2012-06-25 21:36:11 +0000 +++ lava_dispatcher/actions/android_deploy.py 2012-07-06 01:43:50 +0000 @@ -33,14 +33,13 @@ 'system': {'type': 'string'}, 'data': {'type': 'string'}, 'pkg': {'type': 'string', 'optional': True}, - 'use_cache': {'type': 'bool', 'optional': True, 'default': True}, 'rootfstype': {'type': 'string', 'optional': True, 'default': 'ext4'}, }, 'additionalProperties': False, } - def run(self, boot, system, data, pkg=None, use_cache=True, rootfstype='ext4'): + def run(self, boot, system, data, pkg=None, rootfstype='ext4'): if not isinstance(self.client, LavaMasterImageClient) and \ not isinstance(self.client, LavaFastModelClient): raise RuntimeError("Invalid LavaClient for this action") - self.client.deploy_linaro_android(boot, system, data, pkg, use_cache, rootfstype) + self.client.deploy_linaro_android(boot, system, data, pkg, rootfstype) === modified file 'lava_dispatcher/actions/deploy.py' --- lava_dispatcher/actions/deploy.py 2012-06-15 20:36:11 +0000 +++ lava_dispatcher/actions/deploy.py 2012-07-06 01:43:50 +0000 @@ -43,7 +43,6 @@ ## 'hwpack': {'type': 'string'}, ## 'rootfs': {'type': 'string'}, ## 'kernel_matrix': {'type': 'string', 'optional': True}, - ## 'use_cache': {'type': 'bool', 'optional': True, 'default': True}, ## 'rootfstype': {'type': 'string', 'optional': True, 'default': 'ext3'}, ## }, ## 'additionalProperties': False, @@ -58,7 +57,6 @@ 'rootfs': {'type': 'string', 'optional': True}, 'image': {'type': 'string', 'optional': True}, 'kernel_matrix': {'type': 'string', 'optional': True}, - 'use_cache': {'type': 'bool', 'optional': True}, 'rootfstype': {'type': 'string', 'optional': True}, }, 'additionalProperties': False, @@ -78,12 +76,11 @@ raise ValueError('cannot specify kernel_matrix with an image') def run(self, hwpack=None, rootfs=None, image=None, kernel_matrix=None, - use_cache=True, rootfstype='ext3'): + rootfstype='ext3'): if not isinstance(self.client, LavaMasterImageClient) and \ not isinstance(self.client, LavaQEMUClient): raise RuntimeError("Invalid LavaClient for this action") self.client.deploy_linaro( hwpack=hwpack, rootfs=rootfs, image=image, - kernel_matrix=kernel_matrix, use_cache=use_cache, - rootfstype=rootfstype) + kernel_matrix=kernel_matrix, rootfstype=rootfstype) === modified file 'lava_dispatcher/client/fastmodel.py' --- lava_dispatcher/client/fastmodel.py 2012-06-25 21:36:11 +0000 +++ lava_dispatcher/client/fastmodel.py 2012-06-29 01:26:45 +0000 @@ -94,7 +94,7 @@ self._customize_ubuntu() def deploy_linaro_android(self, boot, system, data, pkg=None, - use_cache=True, rootfstype='ext4'): + rootfstype='ext4'): logging.info("Deploying Android on %s" % self.hostname) self._boot = download_image(boot, self.context, decompress=False) === modified file 'lava_dispatcher/client/lmc_utils.py' --- lava_dispatcher/client/lmc_utils.py 2012-06-26 03:59:26 +0000 +++ lava_dispatcher/client/lmc_utils.py 2012-06-29 01:26:45 +0000 @@ -7,7 +7,6 @@ import shutil from tempfile import mkdtemp import sys -import time from lava_dispatcher.client.base import CriticalError from lava_dispatcher.utils import ( @@ -15,7 +14,7 @@ logging_system, ) -def refresh_hwpack(client, kernel_matrix, hwpack, use_cache=True): +def refresh_hwpack(client, kernel_matrix, hwpack): lava_proxy = client.context.lava_proxy LAVA_IMAGE_TMPDIR = client.context.lava_image_tmpdir logging.info("Deploying new kernel") @@ -29,12 +28,8 @@ # download package to local tarball_dir = mkdtemp(dir=LAVA_IMAGE_TMPDIR) os.chmod(tarball_dir, 0755) - if use_cache: - proxy = lava_proxy - else: - proxy = None - kernel_path = download(new_kernel, tarball_dir, proxy) - hwpack_path = download(hwpack, tarball_dir, proxy) + kernel_path = download(new_kernel, tarball_dir, lava_proxy) + hwpack_path = download(hwpack, tarball_dir, lava_proxy) cmd = ("sudo linaro-hwpack-replace -t %s -p %s -r %s" % (hwpack_path, kernel_path, deb_prefix)) @@ -56,7 +51,7 @@ return new_hwpack_path -def generate_image(client, hwpack_url, rootfs_url, kernel_matrix, use_cache=True, rootfstype=None): +def generate_image(client, hwpack_url, rootfs_url, kernel_matrix, rootfstype=None): """Generate image from a hwpack and rootfs url :param hwpack_url: url of the Linaro hwpack to download @@ -70,7 +65,7 @@ logging.info(" rootfs: %s" % rootfs_url) if kernel_matrix: logging.info(" package: %s" % kernel_matrix[0]) - hwpack_url = refresh_hwpack(kernel_matrix, hwpack_url, use_cache) + hwpack_url = refresh_hwpack(kernel_matrix, hwpack_url) #make new hwpack downloadable hwpack_url = hwpack_url.replace(LAVA_IMAGE_TMPDIR, '') hwpack_url = '/'.join(u.strip('/') for u in [ @@ -79,16 +74,11 @@ tarball_dir = mkdtemp(dir=LAVA_IMAGE_TMPDIR) os.chmod(tarball_dir, 0755) #fix me: if url is not http-prefix, copy it to tarball_dir - if use_cache: - proxy = lava_proxy - else: - proxy = None - logging.info("Downloading the %s file" % hwpack_url) - hwpack_path = download(hwpack_url, tarball_dir, proxy) + hwpack_path = download(hwpack_url, tarball_dir, lava_proxy) logging.info("Downloading the %s file" % rootfs_url) - rootfs_path = download(rootfs_url, tarball_dir, proxy) + rootfs_path = download(rootfs_url, tarball_dir, lava_proxy) logging.info("linaro-media-create version information") cmd = "sudo linaro-media-create -v" === modified file 'lava_dispatcher/client/master.py' --- lava_dispatcher/client/master.py 2012-07-02 02:25:41 +0000 +++ lava_dispatcher/client/master.py 2012-07-06 01:47:32 +0000 @@ -460,7 +460,7 @@ return download(url, directory, lava_proxy, lava_cookies) def deploy_linaro(self, hwpack=None, rootfs=None, image=None, - kernel_matrix=None, use_cache=True, rootfstype='ext3'): + kernel_matrix=None, rootfstype='ext3'): LAVA_IMAGE_TMPDIR = self.context.lava_image_tmpdir LAVA_IMAGE_URL = self.context.lava_image_url # validate in parameters @@ -476,12 +476,12 @@ # generate image if needed try: if image is None: - image_file = generate_image(self, hwpack, rootfs, kernel_matrix, use_cache) + image_file = generate_image(self, hwpack, rootfs, kernel_matrix) boot_tgz, root_tgz = self._generate_tarballs(image_file) else: tarball_dir = mkdtemp(dir=LAVA_IMAGE_TMPDIR) os.chmod(tarball_dir, 0755) - if use_cache: + if self.context.job_data.get('health_check', False): lava_cachedir = self.context.lava_cachedir if self._are_tarballs_cached(image, lava_cachedir): logging.info("Reusing cached tarballs") @@ -551,7 +551,7 @@ finally: shutil.rmtree(os.path.dirname(boot_tgz)) - def deploy_linaro_android(self, boot, system, data, pkg=None, use_cache=True, rootfstype='ext4'): + def deploy_linaro_android(self, boot, system, data, pkg=None, rootfstype='ext4'): LAVA_IMAGE_TMPDIR = self.context.lava_image_tmpdir LAVA_IMAGE_URL = self.context.lava_image_url logging.info("Deploying Android on %s" % self.hostname) @@ -573,7 +573,7 @@ try: boot_tbz2, system_tbz2, data_tbz2, pkg_tbz2 = \ - self._download_tarballs(boot, system, data, pkg, use_cache) + self._download_tarballs(boot, system, data, pkg) except: logging.error("Unable to download artifacts for deployment") tb = traceback.format_exc() @@ -614,15 +614,13 @@ shutil.rmtree(self.tarball_dir) logging.info("Android image deployment exiting") - def _download_tarballs(self, boot_url, system_url, data_url, pkg_url=None, - use_cache=True): + def _download_tarballs(self, boot_url, system_url, data_url, pkg_url=None): """Download tarballs from a boot, system and data tarball url :param boot_url: url of the Linaro Android boot tarball to download :param system_url: url of the Linaro Android system tarball to download :param data_url: url of the Linaro Android data tarball to download :param pkg_url: url of the custom kernel tarball to download - :param use_cache: whether or not to use the cached copy (if it exists) """ lava_proxy = self.context.lava_proxy LAVA_IMAGE_TMPDIR = self.context.lava_image_tmpdir @@ -631,13 +629,11 @@ os.chmod(tarball_dir, 0755) logging.info("Downloading the image files") - proxy = lava_proxy if use_cache else None - - boot_path = download(boot_url, tarball_dir, proxy) - system_path = download(system_url, tarball_dir, proxy) - data_path = download(data_url, tarball_dir, proxy) + boot_path = download(boot_url, tarball_dir, lava_proxy) + system_path = download(system_url, tarball_dir, lava_proxy) + data_path = download(data_url, tarball_dir, lava_proxy) if pkg_url: - pkg_path = download(pkg_url, tarball_dir, proxy) + pkg_path = download(pkg_url, tarball_dir, lava_proxy) else: pkg_path = None logging.info("Downloaded the image files") === modified file 'lava_dispatcher/client/qemu.py' --- lava_dispatcher/client/qemu.py 2012-06-15 20:35:05 +0000 +++ lava_dispatcher/client/qemu.py 2012-06-29 01:26:45 +0000 @@ -47,9 +47,9 @@ super(LavaQEMUClient, self).__init__(context, config) self._lava_image = None - def deploy_linaro(self, hwpack=None, rootfs=None, image=None, kernel_matrix=None, use_cache=True, rootfstype='ext3'): + def deploy_linaro(self, hwpack=None, rootfs=None, image=None, kernel_matrix=None, rootfstype='ext3'): if image is None: - image_file = generate_image(self, hwpack, rootfs, kernel_matrix, use_cache, rootfstype) + image_file = generate_image(self, hwpack, rootfs, kernel_matrix, rootfstype) else: image_file = download_image(image, self.context) self._lava_image = image_file