Firefox Performance Testing : A Python framework for Windows Documentation

Framework Prerequisites

You'll need to download these before running performance testing:

• Python 2.4
  • The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
• Python Win32 Extensions
  • These extensions provide some support for process management and performance monitoring.
• Numerical Python
  • This is required by Matplotlib.
• Matplotlib
  • This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
• PySyc
  • This is used for reading/writing config files in YAML.
• Cygwin
  • For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
• msvcp71.dll
  • Microsoft® C Runtime Library, v. 7.10.3077.0
  • You need a copy of this file to run Python with all the libraries.
  • If you have Visual Studio 2003 installed, you've already got it.

Framework Configuration

Configure the environment

1. Make sure the prerequisites are installed.
2. Make a directory for the framework:

   $mkdir C:\mozilla

3. Download entire framework (directories and all subdirectories) and put it in C:\mozilla

Configure the paths (paths.py)

To setup the appropriate paths for the framework, open the paths.py file in c:\mozilla\testing\performance\win32\ and configure the paths:

CYGWIN

• Verify that the path is pointing to the desired cygwin directory

CYGWIN = r'c:\cygwin\bin\bash.exe -c'

• The path to cygwin bash to get output from dump() in Firefox
• By default, the Cygwin you just installed is in your C:\

SYNC

• Verify that the path is pointing to the desired cygwin directory

SYNC = r'c:\cygwin\bin\sync'

• Runs sync between Ts runs
• By default, the Cygwin you just installed is in C:\

BASE_PROFILE_DIR

• Verify that BASE_PROFILE_DIR is pointing to the base_profile directory
• If you didn't alter the directory structure, the base_profile path would be: C:\mozilla\testing\performance\win32\base_profile

BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'

• IMPORTANT: Check that there is content in the base_profile directory

• (Optional) For the page load test to work, modify the hostperm.1 file
  • Set to allow urls with scheme:file to open in new windows
  • Set the preference to open new windows in a tab should be off.

REPORTS_DIR

• This is the path to the directory that the generated reports will go into

REPORTS_DIR = r'c:\C:\mozilla\testing\performance\win32\extension_perf_reports'

• Create extension_perf_reports in C:\mozilla\testing\performance\win32\

OR

• Set REPORT_DIR path to a directory where you want the reports to go into. Make sure that directory is created.

INIT_URL

• The path to the file url to load when initializing a new profile

INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'

• Check that initialize.html is in C:\mozilla\testing\performance\win32\
• Note: Don't change the local file url to a file path

TS_URL

• The path to the file url to load for startup test (Ts)

TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='

• Check that startup_test.html is in C:\mozilla\testing\performance\win32\startup_test
• Note: Don't change the local file url to a file path

TP_URL

• The path to the file url to load for page load test (Tp)

TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'

• Check that cycler.html is in C:\mozilla\testing\performance\win32\page_load_test
• Note: Don't change the local file url to a file path

Config file

In C:\mozilla\testing\performance\win32\, create a YAML config file (config.yaml) with info about the profiles you want to test. Copy, paste and modify the following in config.yaml:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences : {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

• Instructions:

1. Check that your firefox path is right
2. If you don't have preferences to set:

• change to:

  preferences : {}

1. If you don't have extensions to install in test:

• change to:

  extensions: {}

1. filename and title can be anything

5. Run run_tests.py with the name of your config file as an argument in c:\mozilla\testing\performance\win32\:

$run_tests.py config.yaml

• You can use a space-separated list of config files, to generate a report of startup and page load times.
• The report will be placed in the REPORTS_DIR directory you specified in paths.py.

Directory Structure of Framework

page_load_test/

This directory contains the JavaScript files and html data files for the page load test. The page load test opens a new window and cycles through loading each html file, timing each load.

startup_test/

This directory contains the JavaScript to run the startup test. It measures how long it takes Firefox to start up.

base_profile/

This directory contains the base profile used for testing. A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py. For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set. The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.

run_tests.py, paths.py

These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.