Difference between revisions of "Historical Hera Config"

From CDOT Wiki
Jump to: navigation, search
(Roadmap: checkpoint save)
m (Mozilla@Seneca Cluster Administration moved to Historical Hera Config: This Hera Administration wiki is being archived.)
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Mozilla@Seneca Cluster Administration ==
 
 
 
== Description ==
 
== Description ==
  
Line 8: Line 6:
  
 
* [[User:bhearsum|Ben Hearsum]] (bhearsum)
 
* [[User:bhearsum|Ben Hearsum]] (bhearsum)
* [[User:jp|Justin Pacheco]] (jp)
 
  
== Contributor(s) ==
+
== News ==
  
None as of yet.
+
[[User:Bhearsum|bhearsum]] 13:16, 15 November 2006 (EST)
 +
:* A redid the Ubuntu server generic image as an edgy install. Along with this I made notes about how to make a generic install. I will be doing an Ubuntu Desktop install shortly, but I'm waiting on Justin's stuff re: VNC.
  
== Roadmap ==
+
[[User:Bhearsum|bhearsum]] 12:40, 24 November 2006 (EST)
 +
:* The cluster was down for maintenance recently. Some network upgrades were done at that time. Nobody has felt any lag since this happened. The cluster lag is *tentatively* solved.
  
=== <u>Documentation</u> ===
+
[[User:Bhearsum|bhearsum]] 12:16, 13 December 2006 (EST)
 +
:* I'm playing with Xen today. I just rebooted linux15 with a Xen kernel. I hope the machine comes back up!
  
Documentation is extremely important in this project. Because many people have administrative access to the cluster machines soft copy documentation is necessary. Anyone who needs to know something about the cluster '''should''' be able to find it within the documentation.
+
[[User:Bhearsum|bhearsum]] 12:16, 14 December 2006 (EST)
 +
:* I broke linux15!
 +
:* Also, I looked into using Buildbot Locks for the Places installation but the Builder's are not smart enough to build on the idle machine so that's a dead end.
 +
:* I'm still trying to figure out why warnOnFailure and warnOnWarnings do not work. This is important so that we can report "testfailed" to the Tinderbox instead of "busted".
 +
:* I added the [[Mozilla@Seneca Cluster Administration/Steps for Creating a Generic VM|Steps for Creating a Generic VM]] section
  
So far, there are three pages (so far) that must be kept up-to-date: [[Mozilla@Seneca Cluster Administration/Forwarded Ports|Forwarded Ports]], [[Mozilla@Seneca Cluster Administration/List of Cluster Users|List of Cluster Users]], and [[Mozilla@Seneca Cluster Administration/Cluster Machine Tasks|Cluster Machine Tasks]].
+
[[User:David.humphrey|dave]] 4:24, 21 January 2008 (EST)
 +
:* Planning for a reconfig of cluster to buildbot try.  Setup doc is [[Hera Try Server Setup|here]].
  
* Any time that a new port is forwarded the [[Mozilla@Seneca Cluster Administration/Forwarded Ports|Forwarded Ports]] must be updated. This page is mainly used as a reference for users in case they forget what port to connect on for ssh, http, etc.
+
== Roadmap ==
* Any time a new user is given access to a cluster machine or a VM running on a cluster machine the [[Mozilla@Seneca Cluster Administration/List of Cluster Users|List of Cluster Users]] must be updated. This is just a quick reference page.
 
* [[Mozilla@Seneca Cluster Administration/Cluster Machine Tasks|Cluster Machine Tasks]] is the most important of all three. It tracks exactly who has access to what machine or VM, what servers are running on them, and in the case of VMs their IP address.
 
 
 
This is something that '''everyone''' needs to keep in mind when working on the cluster, not just the administrators.
 
  
=== <u>Real World IPs</u> ===
+
=== <u>Short Term</u> ===
  
 +
==== Real World IPs ====
 
This is a high priority item that we do not have direct control over. The people in charge are working to make this happen and hopefully it will happen soon.
 
This is a high priority item that we do not have direct control over. The people in charge are working to make this happen and hopefully it will happen soon.
  
Line 38: Line 40:
 
* We need to find out if ACS needs to be informed when new port forwards are done.
 
* We need to find out if ACS needs to be informed when new port forwards are done.
  
=== <u>Windows and Linux Generic VMs</u> ===
+
==== Windows and Linux Generic VMs ====
 
As the cluster usage becomes higher and higher it is important to have easy and quick ways to bring up new machines. Generic Ubuntu images have been made already but the quality of them is currently unknown. Some testing should be done to make sure they are stable and complete. I would like to have this done before the start of the next Winter 2007 semester as there will probably be an influx of requests for VMs by new DPS909 students.
 
As the cluster usage becomes higher and higher it is important to have easy and quick ways to bring up new machines. Generic Ubuntu images have been made already but the quality of them is currently unknown. Some testing should be done to make sure they are stable and complete. I would like to have this done before the start of the next Winter 2007 semester as there will probably be an influx of requests for VMs by new DPS909 students.
  
When the images are created the steps for creating a VM and the steps for bringing up a new VM should be documented. This is listed in the table in the [[#Project Details|project details]] section.
+
When the images are created the steps for creating a VM and the steps for bringing up a new VM should be documented. This is listed in the table in the [[#Details|project details]] section.
 +
 
 +
==== VNC access to Linux machines ====
 +
VNC normally runs on a single port running a single X session. This makes it very tough to have multiple users connected at the same time -- or even multiple users using different accounts. This is an issue that we want solved. When someone connects on VNC it should be the same as sitting down at the desktop. GDM should allow a login to that users' desktop. When a second user connects they should see the same thing. This may have to be done through some XDMCP but it '''must''' but compatible with Windows and Linux VNC clients.
 +
 
 +
==== Look into Networking Management Systems ====
 +
As the cluster is used more and more it will become more difficult to manage by hand. I'd like a network management system such as Nagios to be used to help speed up maintenance.
 +
 
 +
 
 +
=== <u>Medium Term</u> ===
 +
 
 +
==== Implement Network Management System ====
 +
Implement one of the systems described above.
 +
 
 +
 
 +
=== <u>Ongoing</u> ===
 +
 
 +
==== Documentation ====
 +
Documentation is extremely important in this project. Because many people have administrative access to the cluster machines soft copy documentation is necessary. Anyone who needs to know something about the cluster '''should''' be able to find it within the documentation.
 +
 
 +
So far, there are three pages (so far) that must be kept up-to-date: [[Mozilla@Seneca Cluster Administration/Forwarded Ports|Forwarded Ports]], [[Mozilla@Seneca Cluster Administration/List of Cluster Users|List of Cluster Users]], and [[Mozilla@Seneca Cluster Administration/Cluster Machine Tasks|Cluster Machine Tasks]].
 +
 
 +
* Any time that a new port is forwarded the [[Mozilla@Seneca Cluster Administration/Forwarded Ports|Forwarded Ports]] must be updated. This page is mainly used as a reference for users in case they forget what port to connect on for ssh, http, etc.
 +
* Any time a new user is given access to a cluster machine or a VM running on a cluster machine the [[Mozilla@Seneca Cluster Administration/List of Cluster Users|List of Cluster Users]] must be updated. This is just a quick reference page.
 +
* [[Mozilla@Seneca Cluster Administration/Cluster Machine Tasks|Cluster Machine Tasks]] is the most important of all three. It tracks exactly who has access to what machine or VM, what servers are running on them, and in the case of VMs their IP address.
 +
 
 +
This is something that '''everyone''' needs to keep in mind when working on the cluster, not just the administrators.
 +
 
 +
==== High Latency on the Cluster ====
 +
The high latency problem on the cluster seems to be solved but we need to keep an eye on it. If this issues crop up again they should be recognized and reported to ACS very quickly.
  
 
== Details ==
 
== Details ==
Line 49: Line 80:
 
<th style="width: 10%>Priority</th>
 
<th style="width: 10%>Priority</th>
 
<th style="width: 30%">Status</th>
 
<th style="width: 30%">Status</th>
<tr>
 
<td>Test a [[Extending the Buildbot|Buildbot]] TryScheduler with MailNotifier.</td>
 
<td style="background-color: yellow; text-align: center; font-weight: bold; font-size: bigger">Low</td>
 
<td>Getting errors from CVS. Need to e-mail the mailing list again.</td>
 
</tr>
 
 
<tr>
 
<tr>
 
<td>Document all [[List of Cluster Machines|cluster machine tasks]].</td>
 
<td>Document all [[List of Cluster Machines|cluster machine tasks]].</td>
Line 85: Line 111:
 
<tr>
 
<tr>
 
<td>Document the steps involved in making a generic VM. Pre-installed software should be included in this.</td>
 
<td>Document the steps involved in making a generic VM. Pre-installed software should be included in this.</td>
<td style="background-color: #ff671e; text-align: center; font-weight: bold; font-size: bigger">Medium</td>
+
<td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
<td></td>
 
<td></td>
 
</tr>
 
</tr>
Line 100: Line 126:
 
</table>
 
</table>
  
== Related Links ==
+
== Documentation ==
  
 
* [[Mozilla@Seneca Cluster Administration/Cluster Machine Tasks|Cluster Machine Tasks]]
 
* [[Mozilla@Seneca Cluster Administration/Cluster Machine Tasks|Cluster Machine Tasks]]
 
* [[Mozilla@Seneca Cluster Administration/Forwarded Ports|Forwarded Ports]]
 
* [[Mozilla@Seneca Cluster Administration/Forwarded Ports|Forwarded Ports]]
 
* [[Mozilla@Seneca Cluster Administration/List of Cluster Users|List of Cluster Users]]
 
* [[Mozilla@Seneca Cluster Administration/List of Cluster Users|List of Cluster Users]]
 +
* [[Mozilla@Seneca Cluster Administration/Steps for Creating a Generic VM|Steps for Creating a Generic VM]]
 +
 +
== Misc. ==
 
* [[Mozilla@Seneca Cluster Administration/Archive|Archive]]
 
* [[Mozilla@Seneca Cluster Administration/Archive|Archive]]
 
== News ==
 
 
=== November 2006 ===
 
[[User:Bhearsum|bhearsum]] 13:16, 15 November 2006 (EST)
 
:* A redid the Ubuntu server generic image as an edgy install. Along with this I made notes about how to make a generic install. I will be doing an Ubuntu Desktop install shortly, but I'm waiting on Justin's stuff re: VNC.
 
 
[[User:Bhearsum|bhearsum]] 12:40, 24 November 2006 (EST)
 
:* The cluster was down for maintenance recently. Some network upgrades were done at that time. Nobody has felt any lag since this happened. The cluster lag is *tentatively* solved.
 

Latest revision as of 16:58, 1 February 2008

Description

This project page is for all system and network administration tasks on the Mozilla cluster at Seneca.

Leader(s)

News

bhearsum 13:16, 15 November 2006 (EST)

  • A redid the Ubuntu server generic image as an edgy install. Along with this I made notes about how to make a generic install. I will be doing an Ubuntu Desktop install shortly, but I'm waiting on Justin's stuff re: VNC.

bhearsum 12:40, 24 November 2006 (EST)

  • The cluster was down for maintenance recently. Some network upgrades were done at that time. Nobody has felt any lag since this happened. The cluster lag is *tentatively* solved.

bhearsum 12:16, 13 December 2006 (EST)

  • I'm playing with Xen today. I just rebooted linux15 with a Xen kernel. I hope the machine comes back up!

bhearsum 12:16, 14 December 2006 (EST)

  • I broke linux15!
  • Also, I looked into using Buildbot Locks for the Places installation but the Builder's are not smart enough to build on the idle machine so that's a dead end.
  • I'm still trying to figure out why warnOnFailure and warnOnWarnings do not work. This is important so that we can report "testfailed" to the Tinderbox instead of "busted".
  • I added the Steps for Creating a Generic VM section

dave 4:24, 21 January 2008 (EST)

  • Planning for a reconfig of cluster to buildbot try. Setup doc is here.

Roadmap

Short Term

Real World IPs

This is a high priority item that we do not have direct control over. The people in charge are working to make this happen and hopefully it will happen soon.

When we do get them a few things must happen:

  • Firewalls must be put into place on all physical machines. It will be extremely important to lock them down tightly.
  • Port forwards from physical machines to VMs must be done.
  • The Forwarded Ports must be updated with the new port numbers.
  • We need to find out if ACS needs to be informed when new port forwards are done.

Windows and Linux Generic VMs

As the cluster usage becomes higher and higher it is important to have easy and quick ways to bring up new machines. Generic Ubuntu images have been made already but the quality of them is currently unknown. Some testing should be done to make sure they are stable and complete. I would like to have this done before the start of the next Winter 2007 semester as there will probably be an influx of requests for VMs by new DPS909 students.

When the images are created the steps for creating a VM and the steps for bringing up a new VM should be documented. This is listed in the table in the project details section.

VNC access to Linux machines

VNC normally runs on a single port running a single X session. This makes it very tough to have multiple users connected at the same time -- or even multiple users using different accounts. This is an issue that we want solved. When someone connects on VNC it should be the same as sitting down at the desktop. GDM should allow a login to that users' desktop. When a second user connects they should see the same thing. This may have to be done through some XDMCP but it must but compatible with Windows and Linux VNC clients.

Look into Networking Management Systems

As the cluster is used more and more it will become more difficult to manage by hand. I'd like a network management system such as Nagios to be used to help speed up maintenance.


Medium Term

Implement Network Management System

Implement one of the systems described above.


Ongoing

Documentation

Documentation is extremely important in this project. Because many people have administrative access to the cluster machines soft copy documentation is necessary. Anyone who needs to know something about the cluster should be able to find it within the documentation.

So far, there are three pages (so far) that must be kept up-to-date: Forwarded Ports, List of Cluster Users, and Cluster Machine Tasks.

  • Any time that a new port is forwarded the Forwarded Ports must be updated. This page is mainly used as a reference for users in case they forget what port to connect on for ssh, http, etc.
  • Any time a new user is given access to a cluster machine or a VM running on a cluster machine the List of Cluster Users must be updated. This is just a quick reference page.
  • Cluster Machine Tasks is the most important of all three. It tracks exactly who has access to what machine or VM, what servers are running on them, and in the case of VMs their IP address.

This is something that everyone needs to keep in mind when working on the cluster, not just the administrators.

High Latency on the Cluster

The high latency problem on the cluster seems to be solved but we need to keep an eye on it. If this issues crop up again they should be recognized and reported to ACS very quickly.

Details

Task Priority Status
Document all cluster machine tasks. High Ongoing
Create new windows images. Low Waiting for enough things to add to make it worthwhile. So far the following are needed:
  • autoconf
  • ssh
  • wget
Real world IPs for all machines High Waiting on ITT
Investigate high latency on the cluster Low ACS did network upgrades on the cluster. This is tentatively solved.
Create clean images of commonly used machines. Ubuntu, Windows 2003, others? Medium Ubuntu images done. Not sure how to do Windows images yet because of product key issues.
Document the steps involved in making a generic VM. Pre-installed software should be included in this. Done
Document the steps involved in bringing up a new VM. This should cover configuration changes as well as who needs to be notified. Is there a list of things that ACS needs yet? Medium
Find a better way to give VNC access to VMs. Ideally I want GDM running on a vnc port. Medium Justin is currently looking into this.

Documentation

Misc.