Difference between revisions of "Firefox Performance Testing : A Python framework for Windows Documentation"

From CDOT Wiki
Jump to: navigation, search
(Overview of Structure)
(Introduction)
 
(47 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
[[Firefox_Performance_Testing_:_A_Python_framework_for_Windows | Back to Firefox Performance Testing Project]]
 
=Introduction=
 
=Introduction=
 
This documentation is the revised version of Annie Sullivan's [[Readme.txt]] for the Firefox Performance Testing.
 
This documentation is the revised version of Annie Sullivan's [[Readme.txt]] for the Firefox Performance Testing.
 +
 +
Unfortunately, the Firefox Performance Testing is only catered to Windows users at this time.
 +
 +
=Hardware Requirements=
 +
 +
*Win32 system
  
 
=Framework Prerequisites=
 
=Framework Prerequisites=
 +
(List the sequence and provide sh script to automate this)
 
You'll need to download these before running performance testing:
 
You'll need to download these before running performance testing:
 
* [http://www.python.org/ftp/python/2.4/python-2.4.msi Python 2.4]
 
* [http://www.python.org/ftp/python/2.4/python-2.4.msi Python 2.4]
Line 22: Line 30:
 
=Framework Configuration=
 
=Framework Configuration=
 
==Configure the environment==
 
==Configure the environment==
#Make sure the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | prerequisites]] are installed.
+
#Make sure the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | prerequisites]] are installed. (sh script will automate this part - but get user to check)
 
#Make a directory for the framework:<pre>$mkdir C:\mozilla</pre>
 
#Make a directory for the framework:<pre>$mkdir C:\mozilla</pre>
#Download entire framework (directories and all subdirectories) and put it in '''C:\mozilla'''
+
#Download entire [http://www.yorku.ca/abel/liz/mozilla/Win32PerfTesting_v4.zip Performance Testing Framework] (directories and all subdirectories) and put it in '''C:\mozilla'''
 +
(one of two options 1.put reports dir in the framework 2.check if it exists and prompt user if he/she wants it created)
  
 +
==Configure the paths (paths.py)==
 +
(Lots of work in this area.. paths.py is unclear, should specify that it can be modified and that these are just defaults.. and the paths are misleading because people think that they are commands for dos)
  
 
+
To setup the appropriate paths for the framework, open the '''paths.py''' file in '''C:\mozilla\testing\performance\win32\'''  and configure the paths (CYGWIN, SYNC, BASE_PROFILE_DIR, REPORTS_DIR, INIT_URL, TS_URL, TP_URL):  
==Configure the paths (paths.py)==
 
To setup the appropriate paths for the framework, open the '''paths.py''' file in '''C:\mozilla\testing\performance\win32\'''  and configure the paths:  
 
  
  
Line 61: Line 70:
 
*This is the path to the directory that the generated reports will go into
 
*This is the path to the directory that the generated reports will go into
 
<pre>REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'</pre>
 
<pre>REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'</pre>
*Create '''extension_perf_reports''' in '''C:\mozilla\testing\performance\win32\'''
+
*Create '''extension_perf_reports''' dir in '''C:\mozilla\testing\performance\win32\'''
  
 
OR
 
OR
Line 80: Line 89:
 
*Check that '''startup_test.html''' is in '''C:\mozilla\testing\performance\win32\startup_test'''  
 
*Check that '''startup_test.html''' is in '''C:\mozilla\testing\performance\win32\startup_test'''  
 
*'''Note: Don't change the local file url to a file path'''
 
*'''Note: Don't change the local file url to a file path'''
 +
  
 
===TP_URL===  
 
===TP_URL===  
Line 87: Line 97:
 
*'''Note: Don't change the local file url to a file path'''
 
*'''Note: Don't change the local file url to a file path'''
  
 +
=Config file=
  
==Config file==
 
  
 +
==Config info==
 +
(refer to Exended Progress Chart for yaml details - a few things have to be highlighted and clarified)
  
===Config info===
 
 
In '''C:\mozilla\testing\performance\win32\''', create a YAML config file (config.yaml) with info about the profiles you want to test. Copy, paste and '''modify''' the following in config.yaml or refer to [[sample.config]]:
 
In '''C:\mozilla\testing\performance\win32\''', create a YAML config file (config.yaml) with info about the profiles you want to test. Copy, paste and '''modify''' the following in config.yaml or refer to [[sample.config]]:
 
<pre>
 
<pre>
Line 112: Line 123:
 
       # Extensions to install in test (use "extensions: {}" for none)
 
       # Extensions to install in test (use "extensions: {}" for none)
 
       extensions:
 
       extensions:
 +
        # Make sure that the path is directed to the desired location of the extension
 
         # Need quotes around guid because of curly braces
 
         # Need quotes around guid because of curly braces
 
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
 
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
Line 117: Line 129:
 
</pre>
 
</pre>
  
===Instructions:===
+
==Instructions==
1. Check that your firefox path is right
+
1. Verify that the '''firefox''' path is pointing to the right Firefox instance on your system.
 +
* You will hit a zero division error and some other errors if you don't point it to the right path
  
2. If you don't have preferences to set:
+
2. '''preferences''':
*change to: <pre>preferences : {}</pre> (make sure there is a space between ":" and "{" or else it will cause a syntax error)
+
* If you don't have '''preferences''', change to: <pre>preferences : {}</pre> (make sure there is a space between ":" and "{" or else it will cause a syntax error)
 +
* If you do have '''preferences''', refer to [[sample.config]]
  
3. If you don't have extensions to install in test:
+
3. '''extensions''':
*change to: <pre>extensions: {}</pre> (make sure there is a space between ":" and "{" or else it will cause a syntax error)
+
* If you don't have '''extensions''' to install, change to: <pre>extensions: {}</pre> (make sure there is a space between ":" and "{" or else it will cause a syntax error)
 
+
* If you do have '''extensions''' to install, refer to this sample:
4. filename and title can be anything
+
<pre>
 +
      extensions:
 +
        # Make sure that the path is directed to the desired location of the extension
 +
        # Need quotes around guid because of curly braces
 +
        "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
 +
        foo@sample.com : c:\path\to\other\unzipped\xpi
 +
</pre>
 +
* Make sure that the path to the extension is right
  
5. Run '''run_tests.py''' with the name of your config file as an argument in '''c:\mozilla\testing\performance\win32\''': <pre>$run_tests.py config.yaml</pre>
+
4. Change '''filename''' and '''title''' to whatever you desire
  
*You can use a space-separated list of config files, to generate a report of startup and page load times.  
+
5. In '''cmd''', run '''run_tests.py''' with the name of your config file as an argument in '''c:\mozilla\testing\performance\win32\''': <pre>$run_tests.py config.yaml</pre>
*The report will be placed in the '''REPORTS_DIR''' directory you specified in '''paths.py'''.
 
  
 +
* Running the test will close any existing Firefox windows. (this part has to be highlighted)
 +
**You might run into this message on your browser: '''This page should close Firefox. If it does not, please make sure that the dom.allow_scripts_to_close_windows preference is set to true in about:config'''
 +
**Type '''auto:config''' in the address bar and double click on '''dom.allow_scripts_to_close_windows''' to change it to true
 +
* If you see '''processing... Performance Testing is starting''', performance testing is running.. wait..
 +
* After the performance testing is done, your reports will be placed in the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#REPORTS_DIR | '''REPORTS_DIR''']] directory you specified in '''paths.py''' 
 +
* You can use a space-separated list of config files, to generate a report of startup and page load times.
  
 
=Directory Structure of Framework=
 
=Directory Structure of Framework=

Latest revision as of 17:53, 13 December 2006

Back to Firefox Performance Testing Project

Introduction

This documentation is the revised version of Annie Sullivan's Readme.txt for the Firefox Performance Testing.

Unfortunately, the Firefox Performance Testing is only catered to Windows users at this time.

Hardware Requirements

  • Win32 system

Framework Prerequisites

(List the sequence and provide sh script to automate this) You'll need to download these before running performance testing:

  • Python 2.4
    • The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
  • Python Win32 Extensions
    • These extensions provide some support for process management and performance monitoring.
  • Numerical Python
    • This is required by Matplotlib.
  • Matplotlib
    • This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
  • PySyc
    • This is used for reading/writing config files in YAML.
  • Cygwin
    • For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
  • msvcp71.dll(NOTE: If you have Visual Studio 2003 installed, you've already got it.)
    • Microsoft® C Runtime Library, v. 7.10.3077.0
    • You need a copy of this file to run Python with all the libraries.

Framework Configuration

Configure the environment

  1. Make sure the prerequisites are installed. (sh script will automate this part - but get user to check)
  2. Make a directory for the framework:
    $mkdir C:\mozilla
  3. Download entire Performance Testing Framework (directories and all subdirectories) and put it in C:\mozilla

(one of two options 1.put reports dir in the framework 2.check if it exists and prompt user if he/she wants it created)

Configure the paths (paths.py)

(Lots of work in this area.. paths.py is unclear, should specify that it can be modified and that these are just defaults.. and the paths are misleading because people think that they are commands for dos)

To setup the appropriate paths for the framework, open the paths.py file in C:\mozilla\testing\performance\win32\ and configure the paths (CYGWIN, SYNC, BASE_PROFILE_DIR, REPORTS_DIR, INIT_URL, TS_URL, TP_URL):


CYGWIN

  • Verify that the path is pointing to the desired cygwin directory
CYGWIN = r'c:\cygwin\bin\bash.exe -c'
  • The path to cygwin bash to get output from dump() in Firefox
  • If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\


SYNC

  • Verify that the path is pointing to the desired cygwin directory
SYNC = r'c:\cygwin\bin\sync'
  • Runs sync between Ts runs
  • If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\


BASE_PROFILE_DIR

  • Verify that BASE_PROFILE_DIR is pointing to the base_profile directory
  • If you didn't alter the directory structure, the base_profile path would be: C:\mozilla\testing\performance\win32\base_profile
BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'
  • IMPORTANT: Check that there is content in the base_profile directory
  • (Optional) For the page load test to work, modify the hostperm.1 file
    • Set to allow urls with scheme:file to open in new windows
    • Set the preference to open new windows in a tab should be off.


REPORTS_DIR

  • This is the path to the directory that the generated reports will go into
REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'
  • Create extension_perf_reports dir in C:\mozilla\testing\performance\win32\

OR

  • Set REPORT_DIR path to a directory where you want the reports to go into. Make sure that directory is created.


INIT_URL

  • The path to the file url to load when initializing a new profile
INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'
  • Check that initialize.html is in C:\mozilla\testing\performance\win32\
  • Note: Don't change the local file url to a file path


TS_URL

  • The path to the file url to load for startup test (Ts)
TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='
  • Check that startup_test.html is in C:\mozilla\testing\performance\win32\startup_test
  • Note: Don't change the local file url to a file path


TP_URL

  • The path to the file url to load for page load test (Tp)
TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'
  • Check that cycler.html is in C:\mozilla\testing\performance\win32\page_load_test
  • Note: Don't change the local file url to a file path

Config file

Config info

(refer to Exended Progress Chart for yaml details - a few things have to be highlighted and clarified)

In C:\mozilla\testing\performance\win32\, create a YAML config file (config.yaml) with info about the profiles you want to test. Copy, paste and modify the following in config.yaml or refer to sample.config:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences : {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

Instructions

1. Verify that the firefox path is pointing to the right Firefox instance on your system.

  • You will hit a zero division error and some other errors if you don't point it to the right path

2. preferences:

  • If you don't have preferences, change to:
    preferences : {}
    (make sure there is a space between ":" and "{" or else it will cause a syntax error)
  • If you do have preferences, refer to sample.config

3. extensions:

  • If you don't have extensions to install, change to:
    extensions: {}
    (make sure there is a space between ":" and "{" or else it will cause a syntax error)
  • If you do have extensions to install, refer to this sample:
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi 
  • Make sure that the path to the extension is right

4. Change filename and title to whatever you desire

5. In cmd, run run_tests.py with the name of your config file as an argument in c:\mozilla\testing\performance\win32\:
$run_tests.py config.yaml
  • Running the test will close any existing Firefox windows. (this part has to be highlighted)
    • You might run into this message on your browser: This page should close Firefox. If it does not, please make sure that the dom.allow_scripts_to_close_windows preference is set to true in about:config
    • Type auto:config in the address bar and double click on dom.allow_scripts_to_close_windows to change it to true
  • If you see processing... Performance Testing is starting, performance testing is running.. wait..
  • After the performance testing is done, your reports will be placed in the REPORTS_DIR directory you specified in paths.py
  • You can use a space-separated list of config files, to generate a report of startup and page load times.

Directory Structure of Framework

Overview of Structure

A glance at the Framework File Structure (CVS files not included):

      win32
        |
        |
        |__ base_profile (dir)
        |     |
        |     |__ bookmarkbackups (dir)
        |     |    |
        |     |    |__ .html files
        |     |
        |     |__ Cache (dir)
        |     |
        |     |__ .bak, .html, .ini, .dat, .txt, .js, .rdf, .mfl files  
        |             
        |
        |__ page_load_test(dir)
        |     |
        |     |__ base(dir)
        |     |    |
        |     |    |__ other dirs and .html files
        |     |
        |     |__ cycler.html & report.html
        |
        |
        |__ startup_test
        |     |
        |     |__ startup_test.html
        |
        |
        |__ extension_perf_reports (dir for generated reports)
        |
        |
        |__ run_tests.py, paths.py, config.yaml and other .py, .html files
   
  • NOTE: Content in base_profile dir may vary


The following is written by Annie Sullivan (annie.sullivan@gmail.com):

base_profile/

  • This directory contains the base profile used for testing.
  • A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py.
  • For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set.
  • The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.


page_load_test/

  • This directory contains the JavaScript files and html data files for the page load test.
  • The page load test opens a new window and cycles through loading each html file, timing each load.


startup_test/

  • This directory contains the JavaScript to run the startup test.
  • It measures how long it takes Firefox to start up.


extension_perf_report/

  • This directory is where the generated report will go into.
  • You may specify another directory to substitute this directory, but make sure that the paths.py is changed to point to it.


run_tests.py, paths.py

  • These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.