Difference between revisions of "BuildAPI"

From CDOT Wiki
Jump to: navigation, search
Line 1: Line 1:
 
== Project Name ==
 
== Project Name ==
  
BuildAPI
+
'''BuildAPI'''
  
 
== Project Description ==
 
== Project Description ==
 
The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.
 
The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.
 
  
 
Initial contacts: [[User:Armenzg|Armenzg]]
 
Initial contacts: [[User:Armenzg|Armenzg]]
Line 11: Line 10:
  
 
== Project Leader(s) ==
 
== Project Leader(s) ==
 
 
Andrew Singh [http://zenit.senecac.on.ca/wiki/index.php/User:Asingh]
 
Andrew Singh [http://zenit.senecac.on.ca/wiki/index.php/User:Asingh]
  
  
 
== Project Contributor(s) ==
 
== Project Contributor(s) ==
 
 
* [[User:Armenzg|Armenzg]]
 
* [[User:Armenzg|Armenzg]]
  
Line 28: Line 25:
  
  
 
+
''* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.
* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.
 
 
* It will be used for purposes such as:
 
* It will be used for purposes such as:
 
:* identifying critical machines that should not be processing jobs
 
:* identifying critical machines that should not be processing jobs
 
:* discovering rogue slaves
 
:* discovering rogue slaves
:* Generating reports on performance, usage, build times, downtime, and more
+
:* Generating reports on performance, usage, build times, downtime, and more''
  
 
* This project uses Pylons framework:
 
* This project uses Pylons framework:
Line 46: Line 42:
 
** Armen suggested that one student works on the first graph, and the other works on the second graph
 
** Armen suggested that one student works on the first graph, and the other works on the second graph
 
** We will also be working on the documentation
 
** We will also be working on the documentation
 +
  
 
== Project Plan ==
 
== Project Plan ==
Line 54: Line 51:
 
** Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
 
** Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
 
** Create my '''own''' script that can generate a simple graph or to be able to pull information from a database
 
** Create my '''own''' script that can generate a simple graph or to be able to pull information from a database
** '''0.11''' - Complete script to generate a simple graph. Further milestones to follow before 0.2.
+
** '''0.11''' - Complete above script to generate a simple graph. At the moment I am able to pull information from the database. Further milestones to follow before 0.2.
  
 
* 0.2
 
* 0.2
Line 61: Line 58:
 
* 0.3
 
* 0.3
 
** Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.
 
** Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.
 +
  
 
== What Needs to be Learned ==
 
== What Needs to be Learned ==
Line 66: Line 64:
 
* BuildAPI database structure
 
* BuildAPI database structure
 
* How to integrate GoogleAPI or another charting system with Pylons
 
* How to integrate GoogleAPI or another charting system with Pylons
 +
  
 
== Accomplishments ==
 
== Accomplishments ==
Line 73: Line 72:
 
* Able to run in from both regularly and from a python virtual environment (used for testing)
 
* Able to run in from both regularly and from a python virtual environment (used for testing)
 
* Created my own test script using a controller to display information from a selected table in the schedulerdb database
 
* Created my own test script using a controller to display information from a selected table in the schedulerdb database
 +
  
 
== Issues ==
 
== Issues ==
Line 79: Line 79:
 
* Will seek help
 
* Will seek help
 
* '''I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this'''
 
* '''I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this'''
 +
  
 
== Project News ==
 
== Project News ==

Revision as of 07:36, 5 November 2010

Project Name

BuildAPI

Project Description

The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.

Initial contacts: Armenzg


Project Leader(s)

Andrew Singh [1]


Project Contributor(s)

Project Details

  • Updated Objectives (October 28th 2010) from Armenzg
  • We will choose one of the following to do:
    • 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
      • a) This is very useful and could move us forward towards having this information being published publicly for consumption
      • b) I highly encourage this one as understanding the mental model behind it is easier
    • 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)


* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.

  • It will be used for purposes such as:
  • identifying critical machines that should not be processing jobs
  • discovering rogue slaves
  • Generating reports on performance, usage, build times, downtime, and more
  • This project uses Pylons framework:
  • Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.


Project Plan

Goals for each release:

  • 0.1
    • Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
    • Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
    • Create my own script that can generate a simple graph or to be able to pull information from a database
    • 0.11 - Complete above script to generate a simple graph. At the moment I am able to pull information from the database. Further milestones to follow before 0.2.
  • 0.2
    • To be updated shortly
  • 0.3
    • Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.


What Needs to be Learned

  • Pylons Framework
  • BuildAPI database structure
  • How to integrate GoogleAPI or another charting system with Pylons


Accomplishments

  • Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
  • Databases installed from sql dump file
  • Able to run Paste server and BuildAPI pre-existing scripts
  • Able to run in from both regularly and from a python virtual environment (used for testing)
  • Created my own test script using a controller to display information from a selected table in the schedulerdb database


Issues

  • I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
  • Tried regular installation as well as using a python virtual environment
  • Will seek help
  • I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this


Project News

  • October 22nd, 2010
  • October 14, 2010:
    • Meeting of previously named "ScriptFactory" broke into three seperate projects: Mozharness, BuildAPI and Release Simple Bugs -Mustafa
    • We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI
  • Wednesday October 13, 2010:
    • The ScriptFactory group met today with Armen and Ctyler in a conference call
    • We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs

Links