From patchwork Mon Nov 5 08:06:11 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Pigott X-Patchwork-Id: 12662 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id BBC7323E60 for ; Mon, 5 Nov 2012 08:06:14 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id 1F1E4A18FA3 for ; Mon, 5 Nov 2012 08:06:14 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id e10so7323245iej.11 for ; Mon, 05 Nov 2012 00:06:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=aXs8xC8kLBQSRv+9IN+nda1pOCxLR0o5vzpSvRAPaEU=; b=CckVfYlvryNG5cLU2m0qjEbfvFMu/kVtjOqRwDMOG6UAJsprwLTQJiFIemgNfZzstw 6kh/192LmXFuf8WmfeXycMB8Xz816uTkqvr7LEbT8QZf1oARp4PkrJumXbei1HwIFmwZ GcnggYZz5kl49o2c0sSsMbq/twtOKsClfWQzAWdloUDzQIwJoGYlU96vX53u3icAhwlO CsPc+aRzaxHm8ifvGhyH/fyS4aJWQ3Ne0AYZQpeUS9ev6RpmYI2n8xSL8cS+mUEjQl9T ci7jC41WVlBzjjCFAdg02D4f7pC9uwg/dGoq+oAW9VZb/BonlZTsjMDlxXare4BeeBXu vFVA== Received: by 10.50.237.69 with SMTP id va5mr4160956igc.62.1352102773573; Mon, 05 Nov 2012 00:06:13 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.67.148 with SMTP id n20csp168231igt; Mon, 5 Nov 2012 00:06:13 -0800 (PST) Received: by 10.180.24.4 with SMTP id q4mr11943814wif.19.1352102772882; Mon, 05 Nov 2012 00:06:12 -0800 (PST) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id d66si3351800wej.52.2012.11.05.00.06.12 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 05 Nov 2012 00:06:12 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1TVHhD-0006dJ-HF for ; Mon, 05 Nov 2012 08:06:11 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id 69EF4E01BE for ; Mon, 5 Nov 2012 08:06:11 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dispatcher X-Launchpad-Branch: ~linaro-validation/lava-dispatcher/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 434 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dispatcher/trunk] Rev 434: Recast boot solution so that we retry the whole boot loop, rather than just the networking part, ... Message-Id: <20121105080611.1048.65275.launchpad@ackee.canonical.com> Date: Mon, 05 Nov 2012 08:06:11 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="16232"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: a989ba307e2fb4af127441da82bc67be70f8a95f X-Gm-Message-State: ALoCoQmnEYakhIHwoL6WxhjHcVCs/0ez9I/kyjBMHQGl5fGa6gjgsTzVqUWJpF6SqmbPz1b+/y/2 Merge authors: Dave Pigott (dpigott) Related merge proposals: https://code.launchpad.net/~dpigott/lava-dispatcher/retry-whole-boot-loop/+merge/132800 proposed by: Dave Pigott (dpigott) ------------------------------------------------------------ revno: 434 [merge] committer: dave.pigott@linaro.org branch nick: trunk timestamp: Mon 2012-11-05 08:05:38 +0000 message: Recast boot solution so that we retry the whole boot loop, rather than just the networking part, if something goes wrong. modified: lava_dispatcher/device/master.py --- lp:lava-dispatcher https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk You are subscribed to branch lp:lava-dispatcher. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dispatcher/trunk/+edit-subscription === modified file 'lava_dispatcher/device/master.py' --- lava_dispatcher/device/master.py 2012-11-02 11:32:31 +0000 +++ lava_dispatcher/device/master.py 2012-11-04 11:47:31 +0000 @@ -344,42 +344,58 @@ self.proc.expect(self.config.image_boot_msg, timeout=300) self.proc.expect(self.config.master_str, timeout=300) - def do_boot_master(self): - """ - sole boot - just boot the master image and don't do anything else - """ - logging.info("Booting the system master image") - try: - self._soft_reboot() - self._wait_for_master_boot() - except (OperationFailed, pexpect.TIMEOUT) as e: - logging.info("Soft reboot failed: %s" % e) + def boot_master_image(self): + """ + reboot the system, and check that we are in a master shell + """ + attempts = 3 + in_master_image = False + while (attempts > 0) and (not in_master_image): + logging.info("Booting the system master image") try: - self._hard_reboot() + self._soft_reboot() self._wait_for_master_boot() except (OperationFailed, pexpect.TIMEOUT) as e: - msg = "Hard reboot into master image failed: %s" % e - logging.critical(msg) - raise CriticalError(msg) - - - def boot_master_image(self): - """ - reboot the system, and check that we are in a master shell - """ - self.do_boot_master() - self.proc.sendline('export PS1="%s"' % self.MASTER_PS1) - self.proc.expect( - self.MASTER_PS1_PATTERN, timeout=120, lava_no_logging=1) - - runner = MasterCommandRunner(self) - self.master_ip = runner.get_master_ip() - - lava_proxy = self.context.config.lava_proxy - if lava_proxy: - logging.info("Setting up http proxy") - runner.run("export http_proxy=%s" % lava_proxy, timeout=30) - logging.info("System is in master image now") + logging.info("Soft reboot failed: %s" % e) + try: + self._hard_reboot() + self._wait_for_master_boot() + except (OperationFailed, pexpect.TIMEOUT) as e: + msg = "Hard reboot into master image failed: %s" % e + logging.warning(msg) + attempts = attempts - 1 + continue + + try: + self.proc.sendline('export PS1="%s"' % self.MASTER_PS1) + self.proc.expect( + self.MASTER_PS1_PATTERN, timeout=120, lava_no_logging=1) + except pexpect.TIMEOUT as e: + msg = "Failed to get command line prompt: " % e + logging.warning(msg) + attempts = attempts - 1 + continue + + runner = MasterCommandRunner(self) + try: + self.master_ip = runner.get_master_ip() + except NetworkError as e: + msg = "Failed to get network up: " % e + logging.warning(msg) + attempts = attempts - 1 + continue + + lava_proxy = self.context.config.lava_proxy + if lava_proxy: + logging.info("Setting up http proxy") + runner.run("export http_proxy=%s" % lava_proxy, timeout=30) + logging.info("System is in master image now") + in_master_image = True + + if not in_master_image: + msg = "Could not get master image booted properly" + logging.critical(msg) + raise CriticalError(msg) @contextlib.contextmanager def _as_master(self): @@ -462,23 +478,13 @@ def get_master_ip(self): logging.info("Waiting for network to come up") - network_up = False - attempts = 2 - while (attempts <> 0) and (not network_up): - while True: - try: - self.wait_network_up() - except NetworkError: - self._client.do_boot_master() - attempts = attempts - 1 - continue - network_up = True - break - if not network_up: - msg = "Unable to reach LAVA server, check network" + try: + self.wait_network_up(timeout=20) + except NetworkError: + msg = "Unable to reach LAVA server" logging.error(msg) self._client.sio.write(traceback.format_exc()) - raise CriticalError(msg) + raise pattern1 = "<(\d?\d?\d?\.\d?\d?\d?\.\d?\d?\d?\.\d?\d?\d?)>" cmd = ("ifconfig %s | grep 'inet addr' | awk -F: '{print $2}' |"