New Firefox Performance Testing Documentation

Introduction

This documentation is still in progress.

• This document assumes that you have the following:
  • msvcp71.dll (NOTE: If you have Visual Studio 2003 installed, you've already got it.)
    • Microsoft® C Runtime Library, v. 7.10.3077.0
    • You need a copy of this file to run Python with all the libraries.
• Put it in your C: directory if you have to download msvcp71.dll

• This documentation is the revised version of Annie Sullivan's Readme.txt for the Firefox Performance Testing.

• Unfortunately, the Firefox Performance Testing is only catered to Windows users at this time.

Quick start

3 easy steps to run Firefox Performance Testing:

• Install Cygwin
• Save the script
• Run the script

Cygwin Setup

• Cygwin
  • Download cygwin.
  • Install cygwin with the DOS-style line endings.
  • At minimum, you are required to have the following additional tools:
    • unzip (Archive category)
    • wget (Web category)

unzip (Archive category)


wget (Web category)

Note: For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.

Automated Configuration

Save the script

• Save this script at a desired location
  • Script: Script to configure Firefox Performance Testing Framework
• Edit the script to your own choosing. You may provide a list of extensions to install and test in the config.yaml section.
• Note: This script will run the performance testing. Running performance testing will close all opened Firefox browsers! If don't wish for that to happen, simply comment out the last line of the script (to comment out, use #):

#./run_tests.py config.yaml;

• Refer to Run Firefox Performance Tests if you wish to run the performance tests later.

Run the script

• In cygwin, browse to the location where you saved your script.
• Before you run the script, refer to What happens when you run Firefox Performance Tests
• From cygwin, run the script.

For example:

$./perfconfig.sh

What the script does

• Creates directories needed for the performance testing
  • Directory that holds the files for the performance testing
  • Reports directory to hold the results generated
• Downloads and installs pre-requisites
• Write to config.yaml

Config File (config.yaml)

Config File Info

When you run the perfconfig.sh script, a config.yaml file will be written to C:\mozilla\testing\performance\win32\ by default.

Your config.yaml should consist of these items.

• filename
• title
• (Test Profile name) - any name
  • firefox
  • preferences
  • extensions

Refer to sample.config:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences: {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

Config File Instructions

1. Verify that the firefox path is pointing to the right Firefox instance on your system.

• You will hit a zero division error and some other errors if you don't point it to the right path

2. preferences:

• If you don't have preferences, change to:

  preferences: {}

  (make sure there is a space between ":" and "{" or else it will cause a syntax error)
• If you do have preferences, refer to sample.config

3. extensions:

• If you don't have extensions to install, change to:

  extensions: {}

  (make sure there is a space between ":" and "{" or else it will cause a syntax error)
• If you do have extensions to install, refer to this sample:

       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi 

• Make sure that the path to the extension is right

4. Change filename and title to whatever you desire

Run Firefox Performance Tests

• By default, the run_tests.py file that you need to execute to run the Firefox Performance Testing is located at c:\mozilla\testing\performance\win32\
• You can use a space-separated list of config files, to generate a report of startup and page load times
• Before you run Firefox Performance Testing, refer to What happens when you run Firefox Performance Tests
• You can run run_tests.py either from the DOS prompt or Cygwin

DOS

$run_tests.py config.yaml <anotherConfig.yaml>

Cygwin

./run_tests.py config.yaml <anotherConfig.yaml>

Things you can configure manually

Configuring the paths (paths.py)

• You may set up the appropriate paths for the framework manually.
• The current paths are set by default
• After you run perfconfig.sh, the paths.py file is located in C:\mozilla\testing\performance\win32\ by default
• The following are the constants in paths.py:
  • CYGWIN
  • SYNC
  • BASE_PROFILE_DIR
  • REPORTS_DIR
  • INIT_URL
  • TS_URL
  • TP_URL

CYGWIN

By default: CYGWIN = r'c:\cygwin\bin\bash.exe -c'

• The path to cygwin bash to get output from dump() in Firefox
• If you didn't specify a different root directory for the Cygwin to be installed, it will be installed in C:\ by default

SYNC

By default: SYNC = r'c:\cygwin\bin\sync'

• Runs sync between Ts runs
• If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\

BASE_PROFILE_DIR

By default: BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'

• IMPORTANT: Check that there is content in the base_profile directory

• (Optional) For the page load test to work, modify the hostperm.1 file
  • Set to allow urls with scheme:file to open in new windows
  • Set the preference to open new windows in a tab should be off.

REPORTS_DIR

By default: REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'

• The perfconfig.sh script creates a extension_perf_reports dir in C:\mozilla\testing\performance\win32\

INIT_URL

By default: INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'

• The path to the file url to load when initializing a new profile
• Note: Don't change the local file url to a file path

TS_URL

By default: TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='

• The path to the file url to load for startup test (Ts)
• Note: Don't change the local file url to a file path

TP_URL

By default: TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'

• The path to the file url to load for page load test (Tp)
• Note: Don't change the local file url to a file path

Configuring Config file

• If you run the perfconfig.sh script, a config.yaml file will be written to C:\mozilla\testing\performance\win32\ config.yaml or refer to sample.config
• You may have as many config files as you need to use for your Performance Testing
• The framework takes as many arguments as you provide when you run run_tests.py
  • Refer to Run Firefox Performance Tests
• For more information, refer to Config File (config.yaml)

What happens when you run Firefox Performance Tests

• All opened browser windows will be closed automatically!
• You might run into this message on your browser: This page should close Firefox. If it does not, please make sure that the dom.allow_scripts_to_close_windows preference is set to true in about:config
  • Type auto:config in the address bar and double click on dom.allow_scripts_to_close_windows to change it to true
• If you see processing... Performance Testing is starting, performance testing is running..
• You can check the progress by the progress bar
• After the performance testing is done, your reports will be placed in the REPORTS_DIR directory you specified in paths.py
  • By default, it your reports will go to: C:\mozilla\testing\performance\win32\extension_perf_reports
• You can use a space-separated list of config files, to generate a report of startup and page load times.
  • Refer to Run Firefox Performance Tests