Code Indexer

From CDOT Wiki
Revision as of 17:53, 5 November 2006 by John64 (talk | contribs) (→‎Pulling CVS)
Jump to: navigation, search

Project Name

Source Code Indexing Service Analysis

Project Description

Mozilla’s source code is enormous—millions of lines of C, C++, JavaScript, Perl, Python, Java, C#, etc. Developers currently use the lxr system to quickly search and browse it on-line: http://lxr.mozilla.org. Mozilla is planning a move from CVS to Subversion for revision control, and at the same time wants to evaluate other source indexing services. Two BSD students are working to setup, document, and test other potential services (e.g., fisheye, opengrok, mxr) on one of the Seneca-Mozilla servers (see below). In each case this requires configuration changes and some scripting to get the services to properly integrate with Mozilla’s other on-line tools. When the test services are installed and synched with the live source tree, Mozilla will point its developers to them and get feedback—the students will help collect and synthesize this feedback.

Project Leader(s)

John Ford (John64)

Project Contributor(s)

Project News and Details

Status

  • I got OpenGrok to server, but unfortunatley it gives me a read error permission, so I am re-do ing the installation. --John64 20:15, 23 October 2006 (EDT)
  • Due to a huge number of assignments I have, this project has stalled until at least reading week. I should have lots of time during or soon after reading week to spend on this! --John64 10:02, 19 October 2006 (EDT)
  • Update: Since OpenGrok doesn't do file version tracking, There is no point in indexing the entire Mozilla code base. I will be pulling down a copy of the Mozilla source tree to index instead. This shouldn't have the intense footprint as an rsync'd Mozilla tree --John64 00:01, 10 October 2006 (EDT)
  • Issue: OpenGrok is very demanding. It has allready filled a 12GB VMware image, and i still dont know how large the index grows. This could be a problem, not because i dont have space, but because this may be too much for the school server. Likely OpenGrok would need a dedicated, reasonibly fast machine. It seems to be a very powerful, extremely well done application but it is missing the ability to associate "blame" from what I see. I also don't know how it takes care of file versioning. This happened when indexing the entire 2.9GB mozilla CVSROOT. If you take a look at the example OpenGrok I link to, you will see that its a really nice application!
  • Switched to a local VM from my P4 --John64 21:44, 4 October 2006 (EDT)
  • Set Java's location in /etc/default/tomcat5 --John64 20:15, 3 October 2006 (EDT)
  • Figured out port for Tomcat5, 8180 private. Setting up port forwarding as we speak using public port 81 Oct 3, 2006
  • Using personal Machine instead of the VM due to it being double NAT-ed, and thus inaccessible from outside the host machine Oct 2, 2006
  • Set up server using Ubuntu 6.06LTS with Linux 2.6.17ck1 Oct 2, 2006
  • RSyncing machine Oct 2, 2006

Candidates

  • LXR/MXR/Bonsai - Not working on setting one up because there is already one
  • Gonzui - Impressive looking thing. This will be my first target to setup due to its simplicity and apparently very powerful nature
  • OpenGrok - As this is becoming my main focus, I am breaking it out into its own page.
  • OpenGrok - This is by far the coolest project I have come across so far. It uses Java Server Pages, something I know nothing about, so lots of reading. This is my favorite so far. Example: OpenSolaris. A good example of how it takes care of versioning is found at Example. I am looking into the "blame" feature required, it seems to track who did the last version of the file.
  • Fisheye - Commercial Solution that is free (as in beer) for open source projects. Before I start to look at it, I would like to exhaust the numerous open source prospects.
  • Hosted by Tigris - I forgot the name, but its hosted by "Tigris" and it looked pretty good. If you know the name, please edit as appropriate
  • Horus - Not really what is needed, but its a nice interface for programming students own code. I will not be actively doing anything to it.
  • Bazaar - I dont really know what this is, it might be what is needed, but it might be something irrelevant.
  • Sourcenav - investigating this project
  • Google Code Search - No way to specify the Mozilla project only

Links

Notes on Accessing Test Server

If you want to access the test server through anything other than port 80, you are going to have to type in the following address into a browser and note the IP address you get in your address bar. This is because I have dynamic DNS. Everything, including the source itself, will be in the http root for easy access to the files. This is not optimal, and will not stay this way once things advance.

Superfind - Will resolve as www.no-ip.com computer, which is why you have to use a browser to get the IP

Questions

Please edit in an answer if you know

Pulling CVS

This code will pull the CVS for the branches specified in @branches 

#!/usr/bin/perl
use strict;
use warnings;

# Pull CVS from the mozilla project server
my $src_root = "/var/mozilla";

# Where is your make executable (Mozilla needs 3.80, nothing else)
my $make = "/home/jhford/oldmake/make";

# Where is your run.sh for opengrok???
my $opengroker = "/var/opengrok/run.sh";

# Branches to be pulled
my @branches = (
  "HEAD", 
  "REFLOW_20061031_BRANCH", 
  "MOZILLA_1_8_0_BRANCH", 
  "MOZILLA_1_8_BRANCH", 
  "MOZILLA_1_7_BRANCH", 
  "AVIARY_1_0_1_20050124_BRANCH"
);

system ("rm -rf ${src_root}/*");

foreach (@branches){
  system("
    mkdir ${src_root}/$_;
    cd ${src_root}/$_;
    cvs -d :pserver:anonymous:anonymous\x40cvs-mirror.mozilla.org:/cvsroot co -r $_ mozilla/client.mk;
    make -f ${src_root}/${_}/mozilla/client.mk checkout MOZ_CO_PROJECT=all;
  ");
}

system ("bash $opengroker");
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Code_Indexer&oldid=5910"