Firefox Performance Testing : A Python framework for Windows Documentation

Framework Prerequisites

You'll need to download some software onto your Windows machine before running performance testing:

• Python 2.4
  • The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
• Python Win32 Extensions
  • These extensions provide some support for process management and performance monitoring.
• Numerical Python
  • This is required by Matplotlib.
• Matplotlib
  • This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
• PySyc
  • This is used for reading/writing config files in YAML.
• Cygwin
  • For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
• msvcp71.dll
  • Microsoft® C Runtime Library, v. 7.10.3077.0
  • You need a copy of this file to run Python with all the libraries.
  • If you have Visual Studio 2003 installed, you've already got it.

Framework Configuration

1. Make sure the prerequisites are installed.
2. Make a directory for the framework:

$mkdir C:\mozilla

1. Download entire framework (directories and all subdirectories) and put it in C:\mozilla
2. To setup the appropriate paths for the framework, open the paths.py file and check if the paths are correct:

• CYGWIN
  • Path to cygwin bash to get output from dump() in Firefox
  • By default, the Cygwin you just installed is in your C:\ drive

• SYNC
  • Run sync between Ts runs
  • By default, the Cygwin you just installed is in your C:\ drive

• BASE_PROFILE_DIR
  • Path to the base profile directory used for testing.
  • Two ways to configure this:

1. Create C:\extension_perf_testing\base_profile, copy all contents of base_profile folder in C:\project\testing\performance\win32\base_profile into C:\extension_perf_testing\base_profile
2. Configure BASE_PROFILE_DIR to reflect the base_profile path (in this case C:\project\testing\performance\win32\base_profile)

• • For the page load test to work, modify the hostperm.1 file, set to allow urls with scheme:file to open in new windows, and the preference to open new windows in a tab should be off.

• REPORTS_DIR
  • Path to the directory that the generated reports will go into
  • Create C:\extension_perf_reports or configure REPORT_DIR path to point to a directory where you want the reports to go into

• INIT_URL
  • Path to the file url to load when initializing a new profile

• TS_URL
  • The path to the file url to load for startup test (Ts)
• TP_URL
  • The path to the file url to load for page load test (Tp)

1. Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: slowmachine
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences : {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

• Note:
  • Check that your firefox path is right
  • Preferences and extensions can be left as none:
    • use "preferences : {}" for no prefs
    • use "extensions: {}" for none

5. Run run_tests.py with the name of your config file as an argument. You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py.

Directory Structure of Framework

page_load_test/

This directory contains the JavaScript files and html data files for the page load test. The page load test opens a new window and cycles through loading each html file, timing each load.

startup_test/

This directory contains the JavaScript to run the startup test. It measures how long it takes Firefox to start up.

base_profile/

This directory contains the base profile used for testing. A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py. For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set. The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.

run_tests.py, paths.py

These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.