Difference between revisions of "Firefox Performance Testing : A Python framework for Windows Documentation"

From CDOT Wiki
Jump to: navigation, search
(Framework Configuration)
Line 20: Line 20:
  
 
=Framework Configuration=
 
=Framework Configuration=
 +
==Configure the environment==
 
#Make sure the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | prerequisites]] are installed.
 
#Make sure the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | prerequisites]] are installed.
#Make a directory for the framework:
+
#Make a directory for the framework:<pre>$mkdir C:\mozilla</pre>
<pre>$mkdir C:\mozilla</pre>
 
 
#Download entire framework (directories and all subdirectories) and put it in '''C:\mozilla'''
 
#Download entire framework (directories and all subdirectories) and put it in '''C:\mozilla'''
#To setup the appropriate paths for the framework, open the '''paths.py''' file and check if the paths are correct:
 
*CYGWIN
 
**Path to cygwin bash to get output from dump() in Firefox
 
**By default, the Cygwin you just installed is in your C:\ drive
 
  
*SYNC
+
==Configure the paths (paths.py)==
**Run sync between Ts runs
+
#To setup the appropriate paths for the framework, open the '''paths.py''' file in '''c:\mozilla\testing\performance\win32\'''  and configure the paths:
**By default, the Cygwin you just installed is in your C:\ drive
+
===CYGWIN===
 +
*Path to cygwin bash to get output from dump() in Firefox
 +
*By default, the Cygwin you just installed is in your C:\ drive
  
*BASE_PROFILE_DIR
+
===SYNC===
**Path to the base profile directory used for testing.
+
*Run sync between Ts runs
**Two ways to configure this:
+
*By default, the Cygwin you just installed is in your C:\ drive
# Create C:\extension_perf_testing\base_profile, copy all contents of '''base_profile''' folder in '''C:\project\testing\performance\win32\base_profile''' into C:\extension_perf_testing\base_profile
 
# Configure BASE_PROFILE_DIR to reflect the base_profile path (in this case C:\project\testing\performance\win32\base_profile)
 
**For the page load test to work, modify the '''hostperm.1''' file, set to allow urls with scheme:file to open in new windows, and the preference to open new windows in a tab should be off.
 
  
*REPORTS_DIR
+
===BASE_PROFILE_DIR===
**Path to the directory that the generated reports will go into
+
*Path to the base profile directory used for testing.
**Create C:\extension_perf_reports or configure REPORT_DIR path to point to a directory where you want the reports to go into
+
*Check that BASE_PROFILE_DIR is pointing to the '''base_profile''' directory
 +
*If you didn't alter the dir structure, it would be: '''C:\mozilla\testing\performance\win32\base_profile'''
 +
*Check there is content in the base_profile directory
 +
*(Optional) For the page load test to work, modify the '''hostperm.1''' file
 +
**Set to allow urls with scheme:file to open in new windows
 +
**Set the preference to open new windows in a tab should be off.
 +
 
 +
==REPORTS_DIR==
 +
*Path to the directory that the generated reports will go into
 +
**Create '''extension_perf_reports''' in '''C:\mozilla\testing\performance\win32\'''
 +
 
 +
OR
 +
 
 +
**Configure REPORT_DIR path to a directory where you want the reports to go into
  
 
*INIT_URL
 
*INIT_URL
 
**Path to the file url to load when initializing a new profile
 
**Path to the file url to load when initializing a new profile
 +
**Check that '''initialize.html''' is in '''C:\mozilla\testing\performance\win32\'''
 +
**Note: Don't change the local file url to a file path 
 +
<pre>INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'</pre>
  
 
*TS_URL
 
*TS_URL
 
**The path to the file url to load for startup test (Ts)
 
**The path to the file url to load for startup test (Ts)
 +
**Check that '''startup_test.html''' is in '''C:\mozilla\testing\performance\win32\startup_test'''
 +
**Note: Don't change the local file url to a file path 
 +
<pre>TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='</pre>
 +
 
*TP_URL  
 
*TP_URL  
**The path to the file url to load for page load test (Tp)   
+
**The path to the file url to load for page load test (Tp)  
 +
**Check that '''cycler.html''' is in '''C:\mozilla\testing\performance\win32\page_load_test'''
 +
**Note: Don't change the local file url to a file path
 +
<pre>TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'</pre>
 +
   
 
#Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml:
 
#Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml:
 
<pre>
 
<pre>
 
     # Filename will be appended to the timestamp in the report filename.
 
     # Filename will be appended to the timestamp in the report filename.
 
     # Use letters and underscores only
 
     # Use letters and underscores only
     filename: slowmachine
+
     filename: yourfilename
 
     # The title of the report
 
     # The title of the report
 
     title: 1 GHz Celeron
 
     title: 1 GHz Celeron
Line 80: Line 99:
 
***use "preferences : {}" for no prefs
 
***use "preferences : {}" for no prefs
 
***use "extensions: {}" for none
 
***use "extensions: {}" for none
 +
**filename and title can be anything
  
5. Run '''run_tests.py''' with the name of your config file as an argument. You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py.
+
5. Run '''run_tests.py''' with the name of your config file as an argument: <pre>$run_tests.py config.yaml</pre>
 
+
*You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py.
  
 
=Directory Structure of Framework=
 
=Directory Structure of Framework=

Revision as of 23:20, 27 November 2006

Framework Prerequisites

You'll need to download these before running performance testing:

  • Python 2.4
    • The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
  • Python Win32 Extensions
    • These extensions provide some support for process management and performance monitoring.
  • Numerical Python
    • This is required by Matplotlib.
  • Matplotlib
    • This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
  • PySyc
    • This is used for reading/writing config files in YAML.
  • Cygwin
    • For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
  • msvcp71.dll
    • Microsoft® C Runtime Library, v. 7.10.3077.0
    • You need a copy of this file to run Python with all the libraries.
    • If you have Visual Studio 2003 installed, you've already got it.


Framework Configuration

Configure the environment

  1. Make sure the prerequisites are installed.
  2. Make a directory for the framework:
    $mkdir C:\mozilla
  3. Download entire framework (directories and all subdirectories) and put it in C:\mozilla

Configure the paths (paths.py)

  1. To setup the appropriate paths for the framework, open the paths.py file in c:\mozilla\testing\performance\win32\ and configure the paths:

CYGWIN

  • Path to cygwin bash to get output from dump() in Firefox
  • By default, the Cygwin you just installed is in your C:\ drive

SYNC

  • Run sync between Ts runs
  • By default, the Cygwin you just installed is in your C:\ drive

BASE_PROFILE_DIR

  • Path to the base profile directory used for testing.
  • Check that BASE_PROFILE_DIR is pointing to the base_profile directory
  • If you didn't alter the dir structure, it would be: C:\mozilla\testing\performance\win32\base_profile
  • Check there is content in the base_profile directory
  • (Optional) For the page load test to work, modify the hostperm.1 file
    • Set to allow urls with scheme:file to open in new windows
    • Set the preference to open new windows in a tab should be off.

REPORTS_DIR

  • Path to the directory that the generated reports will go into
    • Create extension_perf_reports in C:\mozilla\testing\performance\win32\

OR

    • Configure REPORT_DIR path to a directory where you want the reports to go into
  • INIT_URL
    • Path to the file url to load when initializing a new profile
    • Check that initialize.html is in C:\mozilla\testing\performance\win32\
    • Note: Don't change the local file url to a file path
INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'
  • TS_URL
    • The path to the file url to load for startup test (Ts)
    • Check that startup_test.html is in C:\mozilla\testing\performance\win32\startup_test
    • Note: Don't change the local file url to a file path
TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='
  • TP_URL
    • The path to the file url to load for page load test (Tp)
    • Check that cycler.html is in C:\mozilla\testing\performance\win32\page_load_test
    • Note: Don't change the local file url to a file path
TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'
  1. Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml:
     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences : {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi
  • Note:
    • Check that your firefox path is right
    • Preferences and extensions can be left as none:
      • use "preferences : {}" for no prefs
      • use "extensions: {}" for none
    • filename and title can be anything
5. Run run_tests.py with the name of your config file as an argument:
$run_tests.py config.yaml
  • You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py.

Directory Structure of Framework

page_load_test/

This directory contains the JavaScript files and html data files for the page load test. The page load test opens a new window and cycles through loading each html file, timing each load.

startup_test/

This directory contains the JavaScript to run the startup test. It measures how long it takes Firefox to start up.

base_profile/

This directory contains the base profile used for testing. A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py. For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set. The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.

run_tests.py, paths.py

These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.