Difference between revisions of "New Firefox Performance Testing Documentation"

From CDOT Wiki
Jump to: navigation, search
(Additional things you can configure manually)
(Configuring the paths (paths.py))
Line 135: Line 135:
 
*You may set up the appropriate paths for the framework manually.  
 
*You may set up the appropriate paths for the framework manually.  
 
*The current paths are set by default
 
*The current paths are set by default
*The <b>[[paths.py]]</b> file is located in '''C:\mozilla\testing\performance\win32\''' by default
+
*After you run [[perfconfig.sh]], the <b>[[paths.py]]</b> file is located in '''C:\mozilla\testing\performance\win32\''' by default
 
*The following are the constants in [[paths.py]]:
 
*The following are the constants in [[paths.py]]:
 
**[[#CYGWIN | CYGWIN]]
 
**[[#CYGWIN | CYGWIN]]

Revision as of 16:41, 14 December 2006

Introduction

This documentation is still in progress.

  • This document assumes that you have the following:
    • msvcp71.dll (NOTE: If you have Visual Studio 2003 installed, you've already got it.)
      • Microsoft® C Runtime Library, v. 7.10.3077.0
      • You need a copy of this file to run Python with all the libraries.
  • Put it in your C: directory if you have to download msvcp71.dll
  • This documentation is the revised version of Annie Sullivan's Readme.txt for the Firefox Performance Testing.
  • Unfortunately, the Firefox Performance Testing is only catered to Windows users at this time.

Quick start

3 easy steps to run Firefox Performance Testing:

Cygwin Setup

  • Cygwin
    • Download cygwin.
    • Install cygwin with the DOS-style line endings.
    • At minimum, you are required to have the following additional tools:
      • unzip (Archive category)
      • wget (Web category)


unzip (Archive category)
Cygwin.PNG

wget (Web category)
Cygwin2.PNG

Note: For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.

Automated Configuration

Save the script

  • Save this script at a desired location
  • Edit the script to your own choosing. You may provide a list of extensions to install and test in the config.yaml section.
  • Note: This script will run the performance testing. If don't wish for that to happen, simply comment out the last line of the script (to comment out, use #):
#./run_tests.py config.yaml;

Run the script

For example:

$./perfconfig.sh

What the script does

  • Creates directories needed for the performance testing
    • Directory that holds the files for the performance testing
    • Reports directory to hold the results generated
  • Downloads and installs pre-requisites
  • Write to config.yaml

Config File (config.yaml)

Config File Info

When you run the perfconfig.sh script, a config.yaml file will be written to C:\mozilla\testing\performance\win32\ by default.

Your config.yaml should consist of these items.

  • filename
  • title
  • (Test Profile name) - any name
    • firefox
    • preferences
    • extensions

Refer to sample.config:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences: {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

Config File Instructions

1. Verify that the firefox path is pointing to the right Firefox instance on your system.

  • You will hit a zero division error and some other errors if you don't point it to the right path

2. preferences:

  • If you don't have preferences, change to:
    preferences: {}
    (make sure there is a space between ":" and "{" or else it will cause a syntax error)
  • If you do have preferences, refer to sample.config

3. extensions:

  • If you don't have extensions to install, change to:
    extensions: {}
    (make sure there is a space between ":" and "{" or else it will cause a syntax error)
  • If you do have extensions to install, refer to this sample:
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi 
  • Make sure that the path to the extension is right

4. Change filename and title to whatever you desire

Run Firefox Performance Tests

  • By default, the run_tests.py file that you need to execute to run the Firefox Performance Testing is located at c:\mozilla\testing\performance\win32\
  • You can run run_tests.py either from the DOS prompt or Cygwin:


DOS

$run_tests.py config.yaml


Cygwin

./run_tests.py config.yaml

Things you can configure manually

Configuring the paths (paths.py)

CYGWIN

By default: CYGWIN = r'c:\cygwin\bin\bash.exe -c'

  • The path to cygwin bash to get output from dump() in Firefox
  • If you didn't specify a different root directory for the Cygwin to be installed, it will be installed in C:\ by default


SYNC

By default: SYNC = r'c:\cygwin\bin\sync'

  • Runs sync between Ts runs
  • If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\


BASE_PROFILE_DIR

By default: BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'

  • IMPORTANT: Check that there is content in the base_profile directory
  • (Optional) For the page load test to work, modify the hostperm.1 file
    • Set to allow urls with scheme:file to open in new windows
    • Set the preference to open new windows in a tab should be off.


REPORTS_DIR

By default: REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'

  • The perfconfig.sh script creates a extension_perf_reports dir in C:\mozilla\testing\performance\win32\


INIT_URL

By default: INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'

  • The path to the file url to load when initializing a new profile
  • Note: Don't change the local file url to a file path


TS_URL

By default: TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='

  • The path to the file url to load for startup test (Ts)
  • Note: Don't change the local file url to a file path


TP_URL

By default: TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'

  • The path to the file url to load for page load test (Tp)
  • Note: Don't change the local file url to a file path

Configuring Config file

  • If you run the perfconfig.sh script, a config.yaml file will be written to C:\mozilla\testing\performance\win32\ config.yaml or refer to sample.config
  • You may have as many config files as you need to use for your Performance Testing
  • The framework takes as many arguments as you provide when you run run_tests.py:
  • For more information, refer to

What happens when you run Firefox Performance Tests

  • All opened browser windows will be closed automatically!
  • You might run into this message on your browser: This page should close Firefox. If it does not, please make sure that the dom.allow_scripts_to_close_windows preference is set to true in about:config
    • Type auto:config in the address bar and double click on dom.allow_scripts_to_close_windows to change it to true
  • If you see processing... Performance Testing is starting, performance testing is running..
  • You can check the progress by the progress bar