Firefox Performance Testing : A Python framework for Windows Documentation

From CDOT Wiki
Revision as of 00:30, 29 November 2006 by Elichak (talk | contribs) (Configure the environment)
Jump to: navigation, search

Back to Firefox Performance Testing Project

Introduction

This documentation is the revised version of Annie Sullivan's Readme.txt for the Firefox Performance Testing.

Hardware Requirements

  • Win32 system

Framework Prerequisites

You'll need to download these before running performance testing:

  • Python 2.4
    • The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
  • Python Win32 Extensions
    • These extensions provide some support for process management and performance monitoring.
  • Numerical Python
    • This is required by Matplotlib.
  • Matplotlib
    • This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
  • PySyc
    • This is used for reading/writing config files in YAML.
  • Cygwin
    • For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
  • msvcp71.dll(NOTE: If you have Visual Studio 2003 installed, you've already got it.)
    • Microsoft® C Runtime Library, v. 7.10.3077.0
    • You need a copy of this file to run Python with all the libraries.

Framework Configuration

Configure the environment

  1. Make sure the prerequisites are installed.
  2. Make a directory for the framework:
    $mkdir C:\mozilla
  3. Download entire Performance Testing Framework (directories and all subdirectories) and put it in C:\mozilla

Configure the paths (paths.py)

To setup the appropriate paths for the framework, open the paths.py file in C:\mozilla\testing\performance\win32\ and configure the paths (CYGWIN, SYNC, BASE_PROFILE_DIR, REPORTS_DIR, INIT_URL, TS_URL, TP_URL):


CYGWIN

  • Verify that the path is pointing to the desired cygwin directory
CYGWIN = r'c:\cygwin\bin\bash.exe -c'
  • The path to cygwin bash to get output from dump() in Firefox
  • If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\


SYNC

  • Verify that the path is pointing to the desired cygwin directory
SYNC = r'c:\cygwin\bin\sync'
  • Runs sync between Ts runs
  • If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\


BASE_PROFILE_DIR

  • Verify that BASE_PROFILE_DIR is pointing to the base_profile directory
  • If you didn't alter the directory structure, the base_profile path would be: C:\mozilla\testing\performance\win32\base_profile
BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'
  • IMPORTANT: Check that there is content in the base_profile directory
  • (Optional) For the page load test to work, modify the hostperm.1 file
    • Set to allow urls with scheme:file to open in new windows
    • Set the preference to open new windows in a tab should be off.


REPORTS_DIR

  • This is the path to the directory that the generated reports will go into
REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'
  • Create extension_perf_reports dir in C:\mozilla\testing\performance\win32\

OR

  • Set REPORT_DIR path to a directory where you want the reports to go into. Make sure that directory is created.


INIT_URL

  • The path to the file url to load when initializing a new profile
INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'
  • Check that initialize.html is in C:\mozilla\testing\performance\win32\
  • Note: Don't change the local file url to a file path


TS_URL

  • The path to the file url to load for startup test (Ts)
TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='
  • Check that startup_test.html is in C:\mozilla\testing\performance\win32\startup_test
  • Note: Don't change the local file url to a file path


TP_URL

  • The path to the file url to load for page load test (Tp)
TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'
  • Check that cycler.html is in C:\mozilla\testing\performance\win32\page_load_test
  • Note: Don't change the local file url to a file path

Config file

Config info

In C:\mozilla\testing\performance\win32\, create a YAML config file (config.yaml) with info about the profiles you want to test. Copy, paste and modify the following in config.yaml or refer to sample.config:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences : {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

Instructions

1. Verify that the firefox path is pointing to the right Firefox instance on your system.

  • You will hit a zero division error and some other errors if you don't point it to the right path

2. preferences:

  • If you don't have preferences, change to:
    preferences : {}
    (make sure there is a space between ":" and "{" or else it will cause a syntax error)
  • If you do have preferences, refer to sample.config

3. extensions:

  • If you don't have extensions to install, change to:
    extensions: {}
    (make sure there is a space between ":" and "{" or else it will cause a syntax error)
  • If you do have extensions to install, refer to this sample:
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi 
  • Make sure that the path to the extension is right

4. Change filename and title to whatever you desire

5. In cmd, run run_tests.py with the name of your config file as an argument in c:\mozilla\testing\performance\win32\:
$run_tests.py config.yaml
  • Running the test will close any existing Firefox windows.
    • You might run into this message on your browser: This page should close Firefox. If it does not, please make sure that the dom.allow_scripts_to_close_windows preference is set to true in about:config
    • Type auto:config in the address bar and double click on dom.allow_scripts_to_close_windows to change it to true
  • If you see processing... Performance Testing is starting, performance testing is running.. wait..
  • After the performance testing is done, your reports will be placed in the REPORTS_DIR directory you specified in paths.py
  • You can use a space-separated list of config files, to generate a report of startup and page load times.

Directory Structure of Framework

Overview of Structure

A glance at the Framework File Structure (CVS files not included):

      win32
        |
        |
        |__ base_profile (dir)
        |     |
        |     |__ bookmarkbackups (dir)
        |     |    |
        |     |    |__ .html files
        |     |
        |     |__ Cache (dir)
        |     |
        |     |__ .bak, .html, .ini, .dat, .txt, .js, .rdf, .mfl files  
        |             
        |
        |__ page_load_test(dir)
        |     |
        |     |__ base(dir)
        |     |    |
        |     |    |__ other dirs and .html files
        |     |
        |     |__ cycler.html & report.html
        |
        |
        |__ startup_test
        |     |
        |     |__ startup_test.html
        |
        |
        |__ extension_perf_reports (dir for generated reports)
        |
        |
        |__ run_tests.py, paths.py, config.yaml and other .py, .html files
   
  • NOTE: Content in base_profile dir may vary


The following is written by Annie Sullivan (annie.sullivan@gmail.com):

base_profile/

  • This directory contains the base profile used for testing.
  • A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py.
  • For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set.
  • The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.


page_load_test/

  • This directory contains the JavaScript files and html data files for the page load test.
  • The page load test opens a new window and cycles through loading each html file, timing each load.


startup_test/

  • This directory contains the JavaScript to run the startup test.
  • It measures how long it takes Firefox to start up.


extension_perf_report/

  • This directory is where the generated report will go into.
  • You may specify another directory to substitute this directory, but make sure that the paths.py is changed to point to it.


run_tests.py, paths.py

  • These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.