Difference between revisions of "Buildbot and EC2"

From CDOT Wiki
Jump to: navigation, search
(Downloads)
(Tasks)
Line 141: Line 141:
 
         <td>Load a script on startup in Linux</td>
 
         <td>Load a script on startup in Linux</td>
 
         <td>Make it possible to have a script load when Linux starts</td>
 
         <td>Make it possible to have a script load when Linux starts</td>
         <td style="background-color: lightblue; text-align: center; font-weight: bold; font-size: bigger">Done</td>
+
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
         <td>Adam Delyea</td>
 
         <td>Adam Delyea</td>
 
         <td></td>
 
         <td></td>

Revision as of 11:00, 18 November 2007

Project Name

Buildbot and EC2

Project Description

The BuildBot is software for automatically building and testing other software. EC2 is Amazon's "Elastic Compute Cloud", a service that lets you run Virtual Machines on Amazon's computing grid using a web API. It would be awesome if BuildBot had an EC2 module, and could spawn Virtual Machines on-demand to perform tasks. Conveniently, BuildBot is written in Python, and there's a Python library for interfacing with EC2.

References: Amazon EC2 Basics For Python Programmers

Project Leader(s)

Adam Delyea

Project Contributor(s)

Project Details

Release 0.1

Goals

  • Add in a new build step to Buildbot
  • Allow Buildbot to connect with Amazon's EC2 service

Downloads

    EC2_build_step_nokeys.py

    Note: You will need to install the boto library located here.


About

Basically, this first version attempts to connect to an EC2 account, and then display the connection information to show that it has connected.

It uses a library called "boto", and this library is used to connect to all of Amazon's Web Services

To use this in a Buildbot configuration file, simply call the connect_to_ec2() function, which takes in as paramaters, an Amazon id key, and an Amazon secret key.

Example:

    connection = connect_to_ec2("73ghf9h38fh", "75vo87rVO*&V%O7o75v7i%i75I&5li7%i75")

Release 0.2

Goals

  • Start New Instances
  • Add new instances to build slave list
  • Start a build on an EC2 instance build slave

Downloads

    EC2BuildSlave.py     *Just put this file into your buildbot installation folder so python can access it
    Master.cfg

About

To use the EC2 service with Buildbot, there needs to be some way for Buildbot to dynamically generate a list of build slaves. These build slaves (or just slaves) are hosted as a virtual machine on the EC2 service.

This will work by extending the BuildSlave class to add in some new functionality for starting a new EC2 instance. So, every time you add a new slave to the slave list, another instance will be spwaned, so that you can spawn as many instances as you think you will need to complete a build.

This new BuildSlave class will start a new instance, which will then proceed to send the connection information of the master to the new instance. How exactly, I'm not exactly sure, because it takes a minute for the virtual instance to start up.

The rest basically takes care of itself.

Still have to think about how to turn the new EC2 instances off after the build is done.


Release 0.3

Goals

Downloads

About

Tasks

Task Details Priority Contributors Status Link(s)
Define new build step Add a new build step to the build master config file. The build step will be in an external file and imported into the build master config file. Done Adam Delyea   Writing New Build Steps
Connect To EC2 using Python Utilize the EC2 python library to connect to an account on the EC2 network. Once connected, display some information about the users account to prove it's actually connecting. Done Adam Delyea  
Start new instances Allow Buildbot to spawn an instance on demand. This could possibly be a function called right before the build slave list is populated, and would pass back an array of build slaves. Done Adam Delyea  
Add new instances to build slave list Allow newly created EC2 instances to be added to the build slave list.

After looking more closely at how the buildslave is created, this goal doesn't make any sense, since this is not really how the buildslave list works.

Cut Adam Delyea  
Start a new build on an EC2 instance Allow Buildbot to successfully start a build using a build slave on an EC2 instance Done Adam Delyea  
Load a script on startup in Linux Make it possible to have a script load when Linux starts Done Adam Delyea  
Testing People needed to download the revisions and test them to make sure it works in different environments. Note, this will require you to have your own EC2 account. Low NEEDED! *** Not Started  
Different Builds Need different projects to try and build, to make sure it works to build projects other than Mozilla Low NEEDED! *** Not Started  

Contributions


*** Would you like to contribute? Well here's an excellent opportunity for you to do just that! Feel free to e-mail me at <awdelyea@learn.senecac.on.ca>


Project Resources

Project News

2007-09-17: Project claimed!

2007-10-11: Found a python EC2 library that has all of the functions I will need for interacting with EC2. Just need to write a custom build step to run these library functions.

2007-10-12: Release v0.1 is complete and available for download.

2007-10-12: Release v0.2 loosely defined. More detail to follow.