From patchwork Fri Apr 19 08:35:17 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Schutt X-Patchwork-Id: 16256 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-vc0-f198.google.com (mail-vc0-f198.google.com [209.85.220.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 08B9423A42 for ; Fri, 19 Apr 2013 08:36:12 +0000 (UTC) Received: by mail-vc0-f198.google.com with SMTP id gd11sf6782892vcb.5 for ; Fri, 19 Apr 2013 01:35:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:x-beenthere:x-received:received-spf:x-received :x-forwarded-to:x-forwarded-for:delivered-to:x-received:received-spf :mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state:x-original-sender :x-original-authentication-results:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe :content-type; bh=53VcHFkjESVll/6Yy+8+K0Z1ceggpn0Hc9HVNgvbOxE=; b=L820NUci/slpNVHsVQCSURyHHbgiUQv+7VyWaeATB5b7QKWN+RQBowJXjDKKgTjq0P i6uDC7aJxcOY322IQ5HeGlp6ah2x8ph0Rai1A52NCXvjcAfS2Gt7FD80lNFv3HAg0Tlv 8SwjhY5F6vSD/q79xYMklwhkVr8yKA8Uq13HfVnvSflXCptQHJ6PG589Y+O8sEjy4bI7 pQ7faSUKReWx7QyMeSV7oFUVtge1sFdcKSSpe6Yy2mhA2FX8RiKCJ0N19DG+b53LHUWK jNd1Ky/JGFCW7ua7AnnbN+xEKMgFhY38Y0Pu+cIC/8EvGZe9x5RE6fGF9y9bpxY4ftW0 ybdw== X-Received: by 10.236.47.136 with SMTP id t8mr7206422yhb.13.1366360532471; Fri, 19 Apr 2013 01:35:32 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.94.78 with SMTP id da14ls1822075qeb.74.gmail; Fri, 19 Apr 2013 01:35:32 -0700 (PDT) X-Received: by 10.220.104.68 with SMTP id n4mr10875925vco.37.1366360532244; Fri, 19 Apr 2013 01:35:32 -0700 (PDT) Received: from mail-vc0-f176.google.com (mail-vc0-f176.google.com [209.85.220.176]) by mx.google.com with ESMTPS id u16si8870275vcj.24.2013.04.19.01.35.32 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 19 Apr 2013 01:35:32 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.176; Received: by mail-vc0-f176.google.com with SMTP id hf12so3500152vcb.21 for ; Fri, 19 Apr 2013 01:35:32 -0700 (PDT) X-Received: by 10.221.0.199 with SMTP id nn7mr10960054vcb.14.1366360531996; Fri, 19 Apr 2013 01:35:31 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.58.127.98 with SMTP id nf2csp1543veb; Fri, 19 Apr 2013 01:35:31 -0700 (PDT) X-Received: by 10.194.92.176 with SMTP id cn16mr5778421wjb.51.1366360530605; Fri, 19 Apr 2013 01:35:30 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id ls4si4994690wjb.96.2013.04.19.01.35.29 (version=TLSv1 cipher=RC4-SHA bits=128/128); Fri, 19 Apr 2013 01:35:30 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1UT6n2-0007rj-TS for ; Fri, 19 Apr 2013 08:35:28 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 3C3D1E3032 for ; Fri, 19 Apr 2013 08:35:17 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 584 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 584: Add highbank support Message-Id: <20130419083517.15355.81166.launchpad@ackee.canonical.com> Date: Fri, 19 Apr 2013 08:35:17 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: list X-Generated-By: Launchpad (canonical.com); Revision="16567"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: 4c4a6afc5e6fce55f90236e8a808d58d4f506b7b X-Gm-Message-State: ALoCoQk9qnNMnWlQSja/zS7O/Km8VzSJYdXSxySAC3H3BTVD7RWkFUHKhknfuSCjq9hlnHwl6GpP X-Original-Sender: noreply@launchpad.net X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Merge authors: Antonio Terceiro (terceiro) Michael Hudson-Doyle (mwhudson) Nicholas Schutt (nick-schutt) Related merge proposals: https://code.launchpad.net/~nick-schutt/lava-dispatcher/nicks-highbank-support/+merge/159678 proposed by: Nicholas Schutt (nick-schutt) review: Approve - Antonio Terceiro (terceiro) ------------------------------------------------------------ revno: 584 [merge] committer: Nick Schutt branch nick: trunk timestamp: Fri 2013-04-19 10:35:03 +0200 message: Add highbank support added: lava_dispatcher/default-config/lava-dispatcher/device-types/highbank.conf lava_dispatcher/device/highbank.py lava_dispatcher/ipmi.py modified: lava_dispatcher/client/base.py lava_dispatcher/client/lmc_utils.py lava_dispatcher/config.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/client/base.py' --- lava_dispatcher/client/base.py 2013-03-27 11:22:07 +0000 +++ lava_dispatcher/client/base.py 2013-04-15 16:15:45 +0000 @@ -154,7 +154,7 @@ lava_server_ip = self._client.context.config.lava_server_ip self.run( "LC_ALL=C ping -W4 -c1 %s" % lava_server_ip, - ["1 received", "0 received", "Network is unreachable"], + ["1 received|1 packets received", "0 received|0 packets received", "Network is unreachable"], timeout=5, failok=True) if self.match_id == 0: return True === modified file 'lava_dispatcher/client/lmc_utils.py' --- lava_dispatcher/client/lmc_utils.py 2013-02-18 03:19:14 +0000 +++ lava_dispatcher/client/lmc_utils.py 2013-04-11 13:05:31 +0000 @@ -15,7 +15,8 @@ ) -def generate_image(client, hwpack_url, rootfs_url, outdir, bootloader='u_boot', rootfstype=None): +def generate_image(client, hwpack_url, rootfs_url, outdir, bootloader='u_boot', rootfstype=None, + extra_boot_args=None, image_size=None): """Generate image from a hwpack and rootfs url :param hwpack_url: url of the Linaro hwpack to download @@ -47,6 +48,10 @@ (client.config.lmc_dev_arg, image_file, rootfs_path, hwpack_path, bootloader)) if rootfstype is not None: cmd += ' --rootfs ' + rootfstype + if image_size is not None: + cmd += ' --image-size ' + image_size + if extra_boot_args is not None: + cmd += ' --extra-boot-args "%s"' % extra_boot_args logging.info("Executing the linaro-media-create command") logging.info(cmd) === modified file 'lava_dispatcher/config.py' --- lava_dispatcher/config.py 2013-04-05 17:19:57 +0000 +++ lava_dispatcher/config.py 2013-04-08 13:29:48 +0000 @@ -104,6 +104,8 @@ default='Press Enter to stop auto boot...') vexpress_usb_mass_storage_device = schema.StringOption(default=None) + ecmeip = schema.StringOption() + class OptionDescriptor(object): def __init__(self, name): self.name = name === added file 'lava_dispatcher/default-config/lava-dispatcher/device-types/highbank.conf' --- lava_dispatcher/default-config/lava-dispatcher/device-types/highbank.conf 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/default-config/lava-dispatcher/device-types/highbank.conf 2013-02-14 03:14:34 +0000 @@ -0,0 +1,2 @@ +client_type = highbank +connection_command = ipmitool -I lanplus -U admin -P admin -H %(ecmeip)s sol activate === added file 'lava_dispatcher/device/highbank.py' --- lava_dispatcher/device/highbank.py 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/device/highbank.py 2013-04-18 16:40:27 +0000 @@ -0,0 +1,286 @@ +# Copyright (C) 2012 Linaro Limited +# +# Author: Michael Hudson-Doyle +# Author: Nicholas Schutt +# +# This file is part of LAVA Dispatcher. +# +# LAVA Dispatcher is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# LAVA Dispatcher is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along +# with this program; if not, see . + +import contextlib +import logging +import os +import pexpect +import time + +from lava_dispatcher import tarballcache + +from lava_dispatcher.device.master import ( + MasterCommandRunner, +) +from lava_dispatcher.device.target import ( + Target +) +from lava_dispatcher.errors import ( + NetworkError, + CriticalError, + OperationFailed, +) +from lava_dispatcher.downloader import ( + download_image, + download_with_retry, + ) +from lava_dispatcher.utils import ( + mk_targz, + rmtree, +) +from lava_dispatcher.client.lmc_utils import ( + generate_image, +) +from lava_dispatcher.ipmi import IpmiPxeBoot + + +class HighbankTarget(Target): + + MASTER_PS1 = 'root@master [rc=$(echo \$?)]# ' + MASTER_PS1_PATTERN = 'root@master \[rc=(\d+)\]# ' + + def __init__(self, context, config): + super(HighbankTarget, self).__init__(context, config) + self.proc = self.context.spawn(self.config.connection_command, timeout=1200) + self.device_version = None + if self.config.ecmeip == None: + msg = "The ecmeip address is not set for this target" + logging.error(msg) + raise CriticalError(msg) + self.bootcontrol = IpmiPxeBoot(context, self.config.ecmeip) + + def get_device_version(self): + return self.device_version + + def power_on(self): + self.bootcontrol.power_on_boot_image() + return self.proc + + def power_off(self, proc): + self.bootcontrol.power_off() + + def deploy_linaro(self, hwpack, rfs, bootloader): + image_file = generate_image(self, hwpack, rfs, self.scratch_dir, bootloader, + extra_boot_args='1', image_size='1G') + self._customize_linux(image_file) + self._deploy_image(image_file, '/dev/sda') + + def deploy_linaro_prebuilt(self, image): + image_file = download_image(image, self.context, self.scratch_dir) + self._customize_linux(image_file) + self._deploy_image(image_file, '/dev/sda') + + def _deploy_image(self, image_file, device): + with self._as_master() as runner: + + # compress the image to reduce the transfer size + if not image_file.endswith('.bz2') and not image_file.endswith('gz'): + os.system('bzip2 -9v ' + image_file) + image_file += '.bz2' + + tmpdir = self.context.config.lava_image_tmpdir + url = self.context.config.lava_image_url + image_file = image_file.replace(tmpdir, '') + image_url = '/'.join(u.strip('/') for u in [url, image_file]) + + build_dir = '/builddir' + image_file_base = build_dir + '/' + '/'.join(image_file.split('/')[-1:]) + + decompression_cmd = None + if image_file_base.endswith('.gz'): + decompression_cmd = '/bin/gzip -dc' + elif image_file_base.endswith('.bz2'): + decompression_cmd = '/bin/bzip2 -dc' + + runner.run('mkdir %s' % build_dir) + runner.run('mount -t tmpfs -o size=100%% tmpfs %s' % build_dir) + runner.run('wget -O %s %s' % (image_file_base, image_url), timeout=1800) + + if decompression_cmd != None: + cmd = '%s %s | dd bs=4M of=%s' % (decompression_cmd, image_file_base, device) + else: + cmd = 'dd bs=4M if=%s of=%s' % (image_file_base, device) + + runner.run(cmd, timeout=1800) + runner.run('umount %s' % build_dir) + + self.resize_rootfs_partition(runner) + + def get_partition(self, runner, partition): + if partition == self.config.boot_part: + partition = '/dev/disk/by-label/boot' + elif partition == self.config.root_part: + partition = '/dev/disk/by-label/rootfs' + else: + raise RuntimeError( + 'unknown master image partition(%d)' % partition) + return partition + + def resize_rootfs_partition(self, runner): + partno = '2' + start = None + + runner.run('parted -s /dev/sda print', + response='\s+%s\s+([0-9.]+.B)\s+\S+\s+\S+\s+primary\s+(\S+)' % partno, + wait_prompt=False) + if runner.match_id != 0: + msg = "Unable to determine rootfs partition" + logging.warning(msg) + else: + start = runner.match.group(1) + parttype = runner.match.group(2) + + if parttype == 'ext2' or parttype == 'ext3' or parttype == 'ext4': + runner.run('parted -s /dev/sda rm %s' % partno) + runner.run('parted -s /dev/sda mkpart primary %s 100%%' % start) + runner.run('resize2fs -f /dev/sda%s' % partno) + elif parttpe == 'brtfs': + logging.warning("resize of btrfs partition not supported") + else: + logging.warning("unknown partition type for resize: %s" % parttype) + + + @contextlib.contextmanager + def file_system(self, partition, directory): + logging.info('attempting to access master filesystem %r:%s' % + (partition, directory)) + + assert directory != '/', "cannot mount entire partition" + + with self._as_master() as runner: + runner.run('mkdir -p /mnt') + partition = self.get_partition(runner, partition) + runner.run('mount %s /mnt' % partition) + try: + targetdir = '/mnt/%s' % directory + runner.run('mkdir -p %s' % targetdir) + + parent_dir, target_name = os.path.split(targetdir) + + runner.run('/bin/tar -cmzf /tmp/fs.tgz -C %s %s' % (parent_dir, target_name)) + runner.run('cd /tmp') # need to be in same dir as fs.tgz + + url_base = runner.start_http_server() + + url = url_base + '/fs.tgz' + logging.info("Fetching url: %s" % url) + tf = download_with_retry(self.context, self.scratch_dir, url, False) + + tfdir = os.path.join(self.scratch_dir, str(time.time())) + + try: + os.mkdir(tfdir) + self.context.run_command('/bin/tar -C %s -xzf %s' % (tfdir, tf)) + yield os.path.join(tfdir, target_name) + + finally: + tf = os.path.join(self.scratch_dir, 'fs.tgz') + mk_targz(tf, tfdir) + rmtree(tfdir) + + # get the last 2 parts of tf, ie "scratchdir/tf.tgz" + tf = '/'.join(tf.split('/')[-2:]) + runner.run('rm -rf %s' % targetdir) + self._target_extract(runner, tf, parent_dir) + + finally: + runner.stop_http_server() + runner.run('umount /mnt') + + def _target_extract(self, runner, tar_file, dest, timeout=-1): + tmpdir = self.context.config.lava_image_tmpdir + url = self.context.config.lava_image_url + tar_file = tar_file.replace(tmpdir, '') + tar_url = '/'.join(u.strip('/') for u in [url, tar_file]) + self._target_extract_url(runner,tar_url,dest,timeout=timeout) + + def _target_extract_url(self, runner, tar_url, dest, timeout=-1): + decompression_cmd = '' + if tar_url.endswith('.gz') or tar_url.endswith('.tgz'): + decompression_cmd = '| /bin/gzip -dc' + elif tar_url.endswith('.bz2'): + decompression_cmd = '| /bin/bzip2 -dc' + elif tar_url.endswith('.tar'): + decompression_cmd = '' + else: + raise RuntimeError('bad file extension: %s' % tar_url) + + runner.run('wget -O - %s %s | /bin/tar -C %s -xmf -' + % (tar_url, decompression_cmd, dest), + timeout=timeout) + + @contextlib.contextmanager + def _as_master(self): + self.bootcontrol.power_on_boot_master() + + # Two reboots seem to be necessary to ensure that pxe boot is used. + # Need to identify the cause and fix it + self.proc.expect("Hit any key to stop autoboot:") + self.proc.sendline('') + self.bootcontrol.power_reset_boot_master() + + self.proc.expect("\(initramfs\)") + self.proc.sendline('export PS1="%s"' % self.MASTER_PS1) + self.proc.expect(self.MASTER_PS1_PATTERN, timeout=180, lava_no_logging=1) + runner = HBMasterCommandRunner(self) + runner.run(". /scripts/functions") + device = "eth0" + runner.run("DEVICE=%s configure_networking" % device) + + self.device_version = runner.get_device_version() + + try: + yield runner + finally: + logging.debug("deploy done") + + +target_class = HighbankTarget + + +class HBMasterCommandRunner(MasterCommandRunner): + """A CommandRunner to use when the target is booted into the master image. + """ + http_pid = None + + def __init__(self, target): + super(HBMasterCommandRunner, self).__init__(target) + + def start_http_server(self): + master_ip = self.get_master_ip() + if self.http_pid != None: + raise OperationFailed("busybox httpd already running with pid %" % self.http_pid) + # busybox produces no output to parse for, so run it in the bg and get its pid + self.run('busybox httpd -f &') + self.run('echo pid:$!:pid',response="pid:(\d+):pid",timeout=10) + if self.match_id != 0: + raise OperationFailed("busybox httpd did not start") + else: + self.http_pid = self.match.group(1) + url_base = "http://%s" % (master_ip) + return url_base + + def stop_http_server(self): + if self.http_pid == None: + raise OperationFailed("busybox httpd not running, but stop_http_server called.") + self.run('kill %s' % self.http_pid) + self.http_pid = None + === added file 'lava_dispatcher/ipmi.py' --- lava_dispatcher/ipmi.py 1970-01-01 00:00:00 +0000 +++ lava_dispatcher/ipmi.py 2013-04-12 10:01:48 +0000 @@ -0,0 +1,85 @@ +# Copyright (C) 2013 Linaro Limited +# +# Authors: +# Antonio Terceiro +# Michael Hudson-Doyle +# +# This file is part of LAVA Dispatcher. +# +# LAVA Dispatcher is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# LAVA Dispatcher is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along +# with this program; if not, see . + + +class IPMITool(object): + """ + This class wraps the ipmitool CLI to provide a convenient object-oriented + API that can be composed into the implementation of devices that can be + managed with IPMI. + """ + + def __init__(self, context, host, ipmitool="ipmitool"): + self.host = host + self.context = context + self.ipmitool = ipmitool + + def __ipmi(self, command): + self.context.run_command( + "%s -H %s -U admin -P admin %s" % ( + self.ipmitool, self.host, command + ) + ) + + def set_to_boot_from_disk(self): + self.__ipmi("chassis bootdev disk") + + def set_to_boot_from_pxe(self): + self.__ipmi("chassis bootdev pxe") + + def power_off(self): + self.__ipmi("chassis power off") + + def power_on(self): + self.__ipmi("chassis power on") + + def reset(self): + self.__ipmi("chassis power reset") + + +class IpmiPxeBoot(object): + """ + This class provides a convenient object-oriented API that can be + used to initiate power on/off and boot device selection for pxe + and disk boot devices using ipmi commands. + """ + + def __init__(self, context, host): + self.ipmitool = IPMITool(context, host) + + def power_on_boot_master(self): + self.ipmitool.set_to_boot_from_pxe() + self.ipmitool.power_on() + self.ipmitool.reset() + + def power_reset_boot_master(self): + self.ipmitool.set_to_boot_from_pxe() + self.ipmitool.reset() + + def power_on_boot_image(self): + self.ipmitool.set_to_boot_from_disk() + self.ipmitool.power_on() + self.ipmitool.reset() + + def power_off(self): + self.ipmitool.power_off() +