Mozilla Metrics Server Collection

Project Name

Project Description

This project basically proposes to recieve the data being collected by the Metrics extension and store it within a database. In the future, possibly use the data and display it in some form.

Project Leader(s)

Simon Jung (simonJ)

Project Contributor(s)

backinblakk - Mac 10.5 Patch test.

Andrew

Project Details

PROJECT SPECIFICATIONS FOR VERSION 0.1

This project will require me to do the following:

Download and patch metrics extension
Compile the metrics extension with Minefield
Intialize the metrics extension using a config file
Send a config file from server to client extension
Recieve data from the metrics extension
Save recieved data to a file

OVERVIEW

On initialization of the metrics extension, the extension pings the server for a config file; which is in the form of an xml. This config file is sent back to the extension as a server response and includes a http header along with the config file. The server response will tell the extension two things - a STATUS 200 to say everything is OK, a Content-Type: application/xml to inform that the data being sent is an xml file. Once the extension recieves an OK, the extension will then take the config file and save it into the users Firefox profile folder. If there is a failure, STATUS 3xx, 4xx, or 5xx, the extension will create a config file that disables the extension. Once all these requirements are met, the extension will start collecting data and can be seen by checking "about:metrics".

If there was any data collected when the timer reaches the upload interval (described in the xml file), the extension will BZIP the xml file and HTTP POST the data to the server with a "Content-Type: application/vnd.mozilla.metrics.bz2" to indicate the type of data being sent. The extension will then wait for a new config file to reset the extension. The reset basically clears the old data and resets the config file. This process will continue in a loop.

SAMPLE DATA

Config file sample

Data collected sample

Server CGI Script

Metrics Patch

Project COMPLICATIONS & SOLUTIONS

There seems to be a link problem when trying to compile the extension in windows xp. Reading the source code, and with the help of luser, I was able to figure out some problems within the code. Minor changes were made to make files in both the "/extension/metrics/build" and "/extensions/metrics/test" folders; Added "USE_STATIC_LIBS=1" so that the compiler knows to use the static library. Will create a patch and post on bugzilla once I'm done.

Echoing a file back to the extension was pretty straightforward; however, receiving the proper data back was more difficult. Initially, I assumed the data being sent back was an xml file but after reading the source code in-depth, I found that the data was actually compressed with BZIP2 (x.bz2). Changes were made to the CGI to bunzip2 the data.

Another problem, similar to problem 2, was that my while loop to read stdin was not receiving all the data properly. With the help of "ctyler" I was able to receive the data in-tact.

PROJECT SPECIFICATIONS FOR VERSION 0.2

Set-up MySql Server and Django
Use Django to create a database infrastructure

OVERVIEW

The basis of my 0.2 release, was to create a database infrastructure for the xml data being sent to the server from the Metrics extension. Once this is done, my 0.3 Release would require me to create a script (or use some other functionality in the framework some guy was trying to explain to me on Django Google Groups) that will receive the data and use the Django models already created to parse the data into each table. Currently, the database infrastructure is done for all the elements I am aware the Metrics extension can send. There have been a few complications trying to figure out what the proper config element is to activate the other modules; however, right now I am able to get data regarding - profile, plugins, extensions, uielements, and bookmarks. The project right now includes the database and an admin page. The admin page is a basic login that shows all the different tables and allows you to view all tables. As well, there are search functionalities to looks for specific keywords. The only problem with this is trying to display all the log information at once because of the relationship bugs in admin "edit_inline" options. Hopefully, there will be away around this for 0.3 release, so I can get fully working administrative pages that can search through all the data.

Currently, I have quite a lot of concerns on whether or not it would be good to create this Server using the Django Framework. I am still awaiting a reply from Alex about this. There are a lot of complications with Django in terms of relationships between Objects; and while those complications aren't so bad for web authoring a blog or casual sites that use databases, I don't think it would be the best solutions for creating a database of this magnitude.

SAMPLE DATA

Metrics Models.py

  from django.db import models

  class User(models.Model):
     clientid = models.CharField(primary_key=True, max_length=22)
     def __unicode__(self):
        return self.clientid
     class Admin:
        pass

  class Bookmark(models.Model):
     name = models.CharField(max_length=20)
     def __unicode__(self):
        return self.name
     class Admin:
        pass

  class CpuArch(models.Model):
     cpu_arch = models.CharField(max_length=20)
     def __unicode__(self):
        return self.cpu_arch
     class Admin:
        pass

  class UIElement(models.Model):
     targetid = models.CharField(max_length=24)
     action = models.CharField(max_length=40)
     def __unicode__(self):
        return self.targetid
     class Admin:
        pass

  class Install(models.Model):
     date = models.IntegerField()
     default = models.BooleanField()
     buildid = models.IntegerField()
     class Admin:
        pass

  class Extension(models.Model):
     id = models.CharField(primary_key=True, max_length=22)
     version = models.CharField(max_length=20)
     def __unicode__(self):
        return self.id
     class Admin:
        pass

  class Plugin(models.Model):
     filename = models.CharField(max_length=24)
     name = models.CharField(max_length=22)
     def __unicode__(self):
        return self.name
     class Admin:
        pass

  class Display(models.Model):
     ysize = models.IntegerField()
     xsize = models.IntegerField()
     screens = models.IntegerField()
     class Admin:
        pass

  class BookmarkCount(models.Model):
     foldercount = models.IntegerField()
     itemcount = models.IntegerField()
     seperatorcount = models.IntegerField()
     livemarkcount = models.IntegerField()
     bookmark = models.ForeignKey(Bookmark)
     class Admin:
        pass

  class Session(models.Model):
     session_number = models.IntegerField()
     time = models.IntegerField()
     class Admin:
        pass

  class UIUsage(models.Model):
     time = models.IntegerField()
     window = models.IntegerField()
     ui_element = models.ForeignKey(UIElement)
     session = models.ForeignKey(Session)
     def __unicode__(self):
        return self.ui_element
     class Admin:
        pass

  class Profile(models.Model):
     session = models.ForeignKey(Session)
     cpu_arch = models.ForeignKey(CpuArch)
     memory_mb = models.IntegerField()
     display = models.ForeignKey(Display)
     install = models.ForeignKey(Install)
     extension = models.ManyToManyField(Extension)
     plugin = models.ManyToManyField(Plugin)
     bookmarkcount = models.ForeignKey(BookmarkCount)
     class Admin:
        pass

  class Log(models.Model):
     user = models.ForeignKey(User)
     version = models.IntegerField()
     profile = models.ForeignKey(Profile)
     class Admin:
        pass

Project COMPLICATIONS & SOLUTIONS

- Bug Ticket 24 This bug doesn't allow any objects that have One to One Relationships to be shown in the admin; there are no solutions because its resolved with WON'T FIX. Only way around it was to use Foreign Keys and restrict it in admin mode.
- Bug Ticket 1939 Once again, another Won't fix bug that won't allow me to use proper Multiple Foreign Keys.
- There was a problem setting up Django which was not copying all the proper files needed to build a project. The only solution that worked for me was to copy all the contents from the trunk/django to Python25\Lib\site-packages.
- Although I can't find the exact bug right now, Django has some complications using Many to Many relationships too. Once again, I resolved it by using Foreign Keys. However, these solutions can only be temporary.

PROJECT SPECIFICATIONS V0.3

Design and Implement a working metrics collector
Design and Implement a Python script that parses data into the database using the Django Framework
Design and Implement Django models for all the data being received by the client.
Create a Admin Login/Page to view the data from the database.

All information can be found on my matrix site: Metrics Collector 0.3 Information

Project News

Thursday, October 18th 2007

Finally compiled a working version of Minefield; however there is a problem compiling the Metrics extension; error LNK2005.

Friday, October 19th 2007

Fixed the link errors with minor changes to the source code (detail above)
Attempted to create a server script that can handle the request from the extension.

Saturday, October 20th 2007

Server script is initializing the extension; however all attempts to read the data being sent to the server has failed. The data seems to be jibberish.

Saturday, October 20th 2007

Created a simple html page to test the server script. The html page basically allows me to send xml docs through a http post.
Tested server script to see if it is getting the right data.

Sunday, October 21st 2007

The server script and extension are finally working together fine.
A config file is being sent back and the data is being recieved.
Trying to build the extension with the new patch in Windows XP

Monday, October 22nd 2007

Finally got the metrics extension built and working with the nightly.
Created a patch for the metrics extension.
Going to submit a patch to bugzilla after I get somone to compile it in linux

Monday, November 19th 2007

Finally finshed all the tutorials, installations, set-ups, fixes

Tuesday, November 20th 2007

Made a basic database schema based off the xml I received from the client

Thursday, November 22nd 2007

Tried to Code the database schema; however, I ran into a lot of problems
Asked a few questions about problems with relationships on irc.freenode.net #django and were lead to certain tickets on bugs and documentation.

Friday, November 23rd 2007

Setup another project
Created new schema

Sunday, November 25th 2007

Got some advice in seneca irc channel and ctyler and came up with a much better database structure with somewhat proper relationships (becuz of bugs)
build the models and server project