From patchwork Thu Jun 21 22:30:20 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Doan X-Patchwork-Id: 9537 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 62A1623F19 for ; Thu, 21 Jun 2012 22:30:26 +0000 (UTC) Received: from mail-yx0-f180.google.com (mail-yx0-f180.google.com [209.85.213.180]) by fiordland.canonical.com (Postfix) with ESMTP id DC75BA1888E for ; Thu, 21 Jun 2012 22:30:25 +0000 (UTC) Received: by yenq6 with SMTP id q6so1129414yen.11 for ; Thu, 21 Jun 2012 15:30:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=vEN/wmHX7L5Hap4gGSxJ/1Vr7ENbaC0yAttUYjxPiXk=; b=d04fm++wLd36KUnGaC+oo4/soXd0oklrLOINn3LTyMvXSR+AH0ue9tt/Hll7eCY6ru ZvIKI2ILdc43cyM6yZ3SeQpPA0ohiJfGYT4OsUMtHH4wSfpTwfdhPYSch6v0BBHzst4Z qK0ygtEtnR8U0urR62JMkBJvAoaSyLVzGGKYxjj2vgmYztjhhEx5c/h73vAwoVNYejfs 4yqTMnB9r1TgzQHjzen7h/aWYnlYLObDuDTFdmXDogugJwblAeBQYY4zla2EeXv0JnNo 7fQJ7lLbI+ktzHhUBj0Pi3ZY0yeAT3DqAQQhbXr1jjDIl8mRSaOmhn4N3w9Y0vlHFKyu xdYg== Received: by 10.50.40.193 with SMTP id z1mr67133igk.0.1340317824993; Thu, 21 Jun 2012 15:30:24 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.24.148 with SMTP id v20csp41985ibb; Thu, 21 Jun 2012 15:30:23 -0700 (PDT) Received: by 10.216.228.232 with SMTP id f82mr16015787weq.211.1340317822237; Thu, 21 Jun 2012 15:30:22 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id m1si41855012wiz.24.2012.06.21.15.30.20 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 21 Jun 2012 15:30:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1ShptM-0006rr-HW for ; Thu, 21 Jun 2012 22:30:20 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 6C8E6E004A for ; Thu, 21 Jun 2012 22:30:20 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 330 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 330: merge fastmodel boot support Message-Id: <20120621223020.12580.33508.launchpad@ackee.canonical.com> Date: Thu, 21 Jun 2012 22:30:20 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15459"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: a7b1841ada955a26ba1c07319d2a6911d6c819c5 X-Gm-Message-State: ALoCoQnU/TbU2GP9UL/TQSlU3mrZ7qsvjZWClmFgP07c8BuJ37Qvr7wzxXZ1JGH6Bxmjd+HY+Vn/ Merge authors: Andy Doan (doanac) Related merge proposals: https://code.launchpad.net/~doanac/lava-dispatcher/fastmodel-boot/+merge/111086 proposed by: Andy Doan (doanac) review: Approve - Michael Hudson-Doyle (mwhudson) ------------------------------------------------------------ revno: 330 [merge] committer: Andy Doan branch nick: lava-dispatcher timestamp: Thu 2012-06-21 17:28:48 -0500 message: merge fastmodel boot support added: lava_dispatcher/actions/fastmodel_deploy.py lava_dispatcher/client/fastmodel.py lava_dispatcher/default-config/lava-dispatcher/device-types/fastmodel.conf lava_dispatcher/downloader.py modified: lava_dispatcher/actions/android_deploy.py lava_dispatcher/actions/deploy.py lava_dispatcher/client/base.py lava_dispatcher/client/master.py lava_dispatcher/client/qemu.py lava_dispatcher/context.py lava_dispatcher/utils.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/actions/android_deploy.py' --- lava_dispatcher/actions/android_deploy.py 2012-03-11 23:37:07 +0000 +++ lava_dispatcher/actions/android_deploy.py 2012-06-15 20:36:11 +0000 @@ -20,6 +20,7 @@ # along with this program; if not, see . from lava_dispatcher.actions import BaseAction +from lava_dispatcher.client.master import LavaMasterImageClient class cmd_deploy_linaro_android_image(BaseAction): @@ -38,4 +39,6 @@ } def run(self, boot, system, data, pkg=None, use_cache=True, rootfstype='ext4'): + if not isinstance(self.client, LavaMasterImageClient): + raise RuntimeError("Invalid LavaClient for this action") self.client.deploy_linaro_android(boot, system, data, pkg, use_cache, rootfstype) === modified file 'lava_dispatcher/actions/deploy.py' --- lava_dispatcher/actions/deploy.py 2012-03-11 23:37:07 +0000 +++ lava_dispatcher/actions/deploy.py 2012-06-15 20:36:11 +0000 @@ -18,6 +18,8 @@ # along with this program; if not, see . from lava_dispatcher.actions import BaseAction +from lava_dispatcher.client.master import LavaMasterImageClient +from lava_dispatcher.client.qemu import LavaQEMUClient class cmd_deploy_linaro_image(BaseAction): @@ -77,6 +79,10 @@ def run(self, hwpack=None, rootfs=None, image=None, kernel_matrix=None, use_cache=True, rootfstype='ext3'): + if not isinstance(self.client, LavaMasterImageClient) and \ + not isinstance(self.client, LavaQEMUClient): + raise RuntimeError("Invalid LavaClient for this action") + self.client.deploy_linaro( hwpack=hwpack, rootfs=rootfs, image=image, kernel_matrix=kernel_matrix, use_cache=use_cache, === added file 'lava_dispatcher/actions/fastmodel_deploy.py' --- lava_dispatcher/actions/fastmodel_deploy.py 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/actions/fastmodel_deploy.py 2012-06-21 04:13:56 +0000 @@ -0,0 +1,41 @@ +# Copyright (C) 2012 Linaro Limited +# +# Author: Andy Doan +# +# This file is part of LAVA Dispatcher. +# +# LAVA Dispatcher is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# LAVA Dispatcher is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +from lava_dispatcher.actions import BaseAction +from lava_dispatcher.client.fastmodel import LavaFastModelClient + + +class cmd_deploy_fastmodel_image(BaseAction): + + parameters_schema = { + 'type': 'object', + 'properties': { + 'image': {'type': 'string', 'optional': False}, + 'axf': {'type': 'string', 'optional': False}, + 'image_type': { + 'type': 'string', 'optional': True, 'default': 'ubuntu', + 'enum': ['android', 'ubuntu']}, + }, + 'additionalProperties': False, + } + + def run(self, image, axf, image_type='ubuntu'): + if not isinstance(self.client, LavaFastModelClient): + raise RuntimeError("Invalid LavaClient for this action") + self.client.deploy_image(image, axf, image_type=='android') === modified file 'lava_dispatcher/client/base.py' --- lava_dispatcher/client/base.py 2012-06-13 07:35:23 +0000 +++ lava_dispatcher/client/base.py 2012-06-21 22:28:48 +0000 @@ -384,9 +384,6 @@ if match_id == 1: raise OperationFailed - def deploy_linaro(self, hwpack, rootfs, kernel_matrix=None, use_cache=True, rootfstype='ext3'): - raise NotImplementedError(self.deploy_linaro) - def setup_proxy(self, prompt_str): lava_proxy = self.context.lava_proxy if lava_proxy: @@ -430,9 +427,6 @@ # Android stuff - def deploy_linaro_android(self, boot, system, data, pkg=None, use_cache=True, rootfstype='ext4'): - raise NotImplementedError(self.deploy_linaro_android) - def boot_linaro_android_image(self): """Reboot the system to the test android image.""" self._boot_linaro_android_image() === added file 'lava_dispatcher/client/fastmodel.py' --- lava_dispatcher/client/fastmodel.py 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/client/fastmodel.py 2012-06-21 04:13:56 +0000 @@ -0,0 +1,157 @@ +# Copyright (C) 2012 Linaro Limited +# +# Author: Andy Doan +# +# This file is part of LAVA Dispatcher. +# +# LAVA Dispatcher is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# LAVA Dispatcher is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along +# with this program; if not, see . + +import atexit +import contextlib +import logging +import os +import pexpect +import threading + +from lava_dispatcher.client.base import ( + CommandRunner, + LavaClient, + ) +from lava_dispatcher.client.lmc_utils import ( + image_partition_mounted, + get_partition_offset, + ) +from lava_dispatcher.downloader import ( + download_image, + ) +from lava_dispatcher.utils import ( + logging_spawn, + logging_system, + ) + + +class LavaFastModelClient(LavaClient): + + PORT_PATTERN = 'terminal_0: Listening for serial connection on port (\d+)' + ANDROID_WALLPAPER = 'system/wallpaper_info.xml' + SYS_PARTITION = 2 + DATA_PARTITION = 5 + + def __init__(self, context, config): + super(LavaFastModelClient, self).__init__(context, config) + self._sim_binary = config.get('simulator_binary', None) + lic_server = config.get('license_server', None) + if not self._sim_binary or not lic_server: + raise RuntimeError("The device type config for this device " + "requires settings for 'simulator_binary' and 'license_server'") + + os.putenv('ARMLMD_LICENSE_FILE', lic_server) + self._sim_proc = None + + def _customize_android(self): + with image_partition_mounted(self._sd_image, self.DATA_PARTITION) as d: + wallpaper = '%s/%s' % (d, self.ANDROID_WALLPAPER) + # delete the android active wallpaper as slows things down + logging_system('sudo rm -f %s' % wallpaper) + + #make sure PS1 is what we expect it to be + with image_partition_mounted(self._sd_image, self.SYS_PARTITION) as d: + logging_system( + 'sudo sh -c \'echo "PS1=%s ">> %s/etc/mkshrc\'' % (self.tester_str, d)) + + def _customize_ubuntu(self): + with image_partition_mounted(self._sd_image, self.root_part) as mntdir: + logging_system('sudo echo linaro > %s/etc/hostname' % mntdir) + + def deploy_image(self, image, axf, is_android=False): + self._axf = download_image(axf, self.context) + self._sd_image = download_image(image, self.context) + + logging.debug("image file is: %s" % self._sd_image) + if is_android: + self._customize_android() + else: + self._customize_ubuntu() + + def _close_sim_proc(self): + self._sim_proc.close(True) + + def _close_serial_proc(self): + self.proc.close(True) + + def _get_sim_cmd(self): + return ("%s -a coretile.cluster0.*=%s " + "-C motherboard.smsc_91c111.enabled=1 " + "-C motherboard.hostbridge.userNetworking=1 " + "-C motherboard.mmc.p_mmc_file=%s " + "-C coretile.cache_state_modelled=0 " + "-C coretile.cluster0.cpu0.semihosting-enable=1 ") % ( + self._sim_binary, self._axf, self._sd_image) + + def _boot_linaro_image(self): + if self.proc is not None: + self.proc.close() + if self._sim_proc is not None: + self._sim_proc.close() + + sim_cmd = self._get_sim_cmd() + + # the simulator proc only has stdout/stderr about the simulator + # we hook up into a telnet port which emulates a serial console + logging.info('launching fastmodel with command %r' % sim_cmd) + self._sim_proc = logging_spawn( + sim_cmd, + logfile=self.sio, + timeout=1200) + atexit.register(self._close_sim_proc) + self._sim_proc.expect(self.PORT_PATTERN, timeout=300) + self._serial_port = self._sim_proc.match.groups()[0] + logging.info('serial console port on: %s' % self._serial_port) + + match = self._sim_proc.expect(["ERROR: License check failed!", + "Simulation is started"]) + if match == 0: + raise RuntimeError("fast model license check failed") + + _pexpect_drain(self._sim_proc).start() + + logging.info('simulator is started connecting to serial port') + self.proc = logging_spawn( + 'telnet localhost %s' % self._serial_port, + logfile=self.sio, + timeout=90) + atexit.register(self._close_serial_proc) + + def _boot_linaro_android_image(self): + ''' booting android or ubuntu style images don't differ for FastModel''' + self._boot_linaro_image() + + def reliable_session(self): + return self.tester_session() + +class _pexpect_drain(threading.Thread): + ''' The simulator process can dump a lot of information to its console. If + don't actively read from it, the pipe will get full and the process will + be blocked. This allows us to keep the pipe empty so the process can run + ''' + def __init__(self, proc): + threading.Thread.__init__(self) + self.proc = proc + self.daemon = True #allows thread to die when main main proc exits + def run(self): + # change simproc's stdout so it doesn't overlap the stdout from our + # serial console logging + self.proc.logfile = open('/dev/null', 'w') + self.proc.interact() === modified file 'lava_dispatcher/client/master.py' --- lava_dispatcher/client/master.py 2012-06-19 04:18:31 +0000 +++ lava_dispatcher/client/master.py 2012-06-21 22:28:48 +0000 @@ -33,8 +33,10 @@ import pexpect import errno +from lava_dispatcher.downloader import ( + download_image, + ) from lava_dispatcher.utils import ( - download_image, logging_spawn, logging_system, string_to_list, === modified file 'lava_dispatcher/client/qemu.py' --- lava_dispatcher/client/qemu.py 2012-06-13 20:46:12 +0000 +++ lava_dispatcher/client/qemu.py 2012-06-15 20:35:05 +0000 @@ -22,7 +22,6 @@ import logging import os import pexpect -from tempfile import mkdtemp from lava_dispatcher.client.base import ( CommandRunner, @@ -32,8 +31,10 @@ generate_image, image_partition_mounted, ) +from lava_dispatcher.downloader import ( + download_image, + ) from lava_dispatcher.utils import ( - download_image, logging_spawn, logging_system, ) === modified file 'lava_dispatcher/context.py' --- lava_dispatcher/context.py 2012-06-07 23:38:28 +0000 +++ lava_dispatcher/context.py 2012-06-15 20:37:40 +0000 @@ -22,6 +22,7 @@ import tempfile from lava_dispatcher.config import get_device_config +from lava_dispatcher.client.fastmodel import LavaFastModelClient from lava_dispatcher.client.master import LavaMasterImageClient from lava_dispatcher.client.qemu import LavaQEMUClient from lava_dispatcher.test_data import LavaTestData @@ -38,10 +39,12 @@ self._client = LavaMasterImageClient(self, device_config) elif client_type == 'qemu': self._client = LavaQEMUClient(self, device_config) + elif client_type == 'fastmodel': + self._client = LavaFastModelClient(self, device_config) else: raise RuntimeError( - "this version of lava-dispatcher only supports master & qemu " - "clients, not %r" % device_config.get('client_type')) + "this version of lava-dispatcher only supports master, qemu, " + "and fastmodel clients, not %r" % device_config.get('client_type')) self.test_data = LavaTestData() self.oob_file = oob_file self._host_result_dir = None === added file 'lava_dispatcher/default-config/lava-dispatcher/device-types/fastmodel.conf' --- lava_dispatcher/default-config/lava-dispatcher/device-types/fastmodel.conf 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/default-config/lava-dispatcher/device-types/fastmodel.conf 2012-06-15 20:37:40 +0000 @@ -0,0 +1,5 @@ +client_type=fastmodel +simulator_binary = /opt/arm/RTSM_A15x14-A7x14_VE/bin/RTSM_VE_Cortex-A15x4-A7x4 + +# The license server must also be specified. eg: +#license_server = 8224@192.168.1.10 === added file 'lava_dispatcher/downloader.py' --- lava_dispatcher/downloader.py 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/downloader.py 2012-06-15 20:35:05 +0000 @@ -0,0 +1,139 @@ +# Copyright (C) 2012 Linaro Limited +# +# Author: Andy Doan +# +# This file is part of LAVA Dispatcher. +# +# LAVA Dispatcher is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# LAVA Dispatcher is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along +# with this program; if not, see . + +import atexit +import bz2 +import contextlib +import logging +import os +import shutil +import subprocess +import urllib2 +import urlparse +import zlib + +from tempfile import mkdtemp + +@contextlib.contextmanager +def _scp_stream(url, proxy=None, cookies=None): + process = None + try: + process = subprocess.Popen( + ['ssh', url.netloc, 'cat', url.path], + shell=False, + stdout=subprocess.PIPE + ) + yield process.stdout + finally: + if process: + process.kill() + +@contextlib.contextmanager +def _http_stream(url, proxy=None, cookies=None): + resp = None + handlers = [] + if proxy: + handlers = [urllib2.ProxyHandler({'http': '%s' % proxy})] + opener = urllib2.build_opener(*handlers) + + if cookies: + opener.addheaders.append(('Cookie', cookies)) + + try: + url = urllib2.quote(url.geturl(), safe=":/") + resp = opener.open(url, timeout=30) + yield resp + finally: + if resp: + resp.close() + +@contextlib.contextmanager +def _file_stream(url, proxy=None, cookies=None): + fd = None + try: + fd = open(url.path, 'rb') + yield fd + finally: + if fd: + fd.close() + +@contextlib.contextmanager +def _decompressor_stream(url, imgdir): + fd = None + decompressor = None + + fname,suffix = _url_to_fname_suffix(url, imgdir) + + if suffix == 'gz': + decompressor = zlib.decompressobj(16+zlib.MAX_WBITS) + elif suffix == 'bz2': + decompressor = bz2.BZ2Decompressor() + else: + fname = '%s.%s' % (fname, suffix) #don't remove the file's real suffix + + def write(buff): + if decompressor: + buff = decompressor.decompress(buff) + fd.write(buff) + + try: + fd = open(fname, 'wb') + yield (write,fname) + finally: + if fd: + fd.close + +def _url_to_fname_suffix(url, path='/tmp'): + filename = os.path.basename(url.path) + parts = filename.split('.') + suffix = parts[-1] + filename = os.path.join(path, '.'.join(parts[:-1])) + return (filename, suffix) + +def download_image(url, context, imgdir=None, delete_on_exit=True): + '''downloads a image that's been compressed as .bz2 or .gz and + decompresses it on the file to the cache directory + ''' + logging.info("Downloading image: %s" % url) + if not imgdir: + imgdir = mkdtemp(dir=context.lava_image_tmpdir) + if delete_on_exit: + atexit.register(shutil.rmtree, imgdir) + + url = urlparse.urlparse(url) + stream = None + if url.scheme == 'scp': + reader = _scp_stream + elif url.scheme == 'http' or url.scheme == 'https': + reader = _http_stream + elif url.scheme == 'file': + reader = _file_stream + else: + raise Exception("Unsupported url protocol scheme: %s" % url.scheme) + + with reader(url, context.lava_proxy, context.lava_cookies) as r: + with _decompressor_stream(url, imgdir) as (writer, fname): + bsize = 32768 + buff = r.read(bsize) + while buff: + writer(buff) + buff = r.read(bsize) + return fname + === modified file 'lava_dispatcher/utils.py' --- lava_dispatcher/utils.py 2012-06-13 20:20:03 +0000 +++ lava_dispatcher/utils.py 2012-06-15 20:35:05 +0000 @@ -23,11 +23,9 @@ import logging import os import shutil -import subprocess import urllib2 import urlparse from shlex import shlex -from tempfile import mkdtemp import pexpect @@ -56,28 +54,6 @@ raise RuntimeError("Could not retrieve %s" % url) return filename -def decompress(image_file): - for suffix, command in [('.gz', 'gunzip'), - ('.xz', 'unxz'), - ('.bz2', 'bunzip2')]: - if image_file.endswith(suffix): - logging.info("Uncompressing %s with %s", image_file, command) - uncompressed_name = image_file[:-len(suffix)] - subprocess.check_call( - [command, '-c', image_file], stdout=open(uncompressed_name, 'w')) - return uncompressed_name - return image_file - -def download_image(url, context, imgdir=None): - ''' common download function to be used by clients. This will download - and decompress the image using LMC_COOKIES and/or LMC_PROXY settings - ''' - logging.info("Downloading image: %s" % url) - if not imgdir: - imgdir = mkdtemp(dir=context.lava_image_tmpdir) - img = download(url, imgdir, context.lava_proxy, context.lava_cookies) - return decompress(img) - def link_or_copy_file(src, dest): try: dir = os.path.dirname(dest)