Difference between revisions of "User:Minooz/RepoSyncProj"

From CDOT Wiki
Jump to: navigation, search
(To Learn)
 
(52 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Repositories' Syncing Project=
+
 
==Mercurial==
+
<big><big>'''Repositories' Syncing'''</big></big>
: [[User:Minooz/RepoSyncProj/Mercurial | Min Mercurial]]
+
==Version Control Tool==
 +
* Basic Tutorial @ http://hginit.com/
 +
* '''comparing popular version control tools''' [http://martinfowler.com/bliki/VersionControlTools.html]
 +
# '''[http://mercurial.selenic.com/ Mercurial]''' My notes on -> [[User:Minooz/RepoSyncProj/Mercurial | Min Mercurial]]
 +
#'''CVS'''
 +
#'''SVN'''
 +
#'''Git'''
  
 
==Continuous Integration System==
 
==Continuous Integration System==
: [http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix CI Feature Matrix]
+
*'''Comparing Different CI Systems''' -> [http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix CI Feature Matrix]
 
+
#'''[http://hudson-ci.org/ Hudson]''' My notes on -> [[User:Minooz/RepoSyncProj/Hudson | Min Hudson]]
*'''Hudosn'''
+
#'''[http://cruisecontrol.sourceforge.net/ Cruise Control]''' My notes on -> [[User:Minooz/RepoSyncProj/CruiseControl | Min Cruise Control]]
: [[User:Minooz/RepoSyncProj/Hudson | Min Hudson]]
+
#'''[http://buildbot.net/trac Buildbot]''' My notes on -> [[User:Minooz/RepoSyncProj/Buildbot | Min Buildbot]]
:[http://hudson-ci.org/ Hudson]
+
#'''[http://continuum.apache.org/ Apache Continuum]'''
:[http://blogs.sun.com/glassfishpodcast/entry/episode_007_interview_with_kohsuke Interview with kosuke]
+
#'''Maven'''
:[http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson Meet+Hudson]
+
#'''[http://www.anthillpro.com/html/products/anthillos/default.html Anthill]'''
:[http://deadlock.netbeans.org/hudson/builds Hudson demo]
 
 
 
:Steps to how to use Hudson
 
:1- Installation
 
::[http://wiki.hudson-ci.org/display/HUDSON/Installing+Hudson+as+a+Windows+service Installation on Windows]
 
::Hudson became self executable[http://weblogs.java.net/blog/kohsuke/archive/2007/02/hudson_became_s.html]
 
::[http://www.youtube.com/watch?v=6k0S4O2PnTc&feature=player_embedded Video Kosuke] With slides @http://marakana.com/forums/java/general/79.html
 
 
 
: 2- Plugins
 
::[http://wiki.hudson-ci.org/display/HUDSON/Mercurial+Plugin Mercurial Plugin]
 
::[http://braindump.dk/tech/2007/05/03/eclipse-plugin-for-hudson/ Eclipse Plugin]
 
::[http://pip.openplans.org/ci-server-step-by-step.html How to Install plugins]
 
::[http://developer.symbian.org/wiki/index.php/Using_Hudson_for_Building Good plugin installation]
 
: 3- Start a Job
 
::[http://wiki.hudson-ci.org/display/HUDSON/Building+a+software+project Building Project]
 
::How to use post build scripts with Hudson using regex in log text[http://wiki.hudson-ci.org/display/HUDSON/Post+build+task]
 
 
 
*'''Apache Continuum'''
 
:http://continuum.apache.org/
 
 
 
*'''Cruise Control'''
 
:http://cruisecontrol.sourceforge.net/
 
:http://cruisecontrol.sourceforge.net/dashboard.html
 
:How to build, deploy, and monitor Java applications: [http://media.pragprog.com/titles/auto/scheduled.pdf Pragmatic Project Automation]
 
:[http://pragprog.com/starter_kit Pragmatic Bookshelf]
 
:[http://cruisecontrol.sourceforge.net/gettingstartedsourcedist.html Getting Started]
 
  
*'''Maven'''
+
==Scripting==
 +
*1- '''[http://ant.apache.org/manual/ Ant]''' My notes on -> [[User:Minooz/RepoSyncProj/Ant | Min Ant]]
 +
:A new ant script is created e.g. ''buildHudson.xml'' that triggers the target(assign1.test) of main build file(build.xml) of the project. See below:
 +
<source lang=java>
 +
<project name="assign1" basedir="." default="myTarget">
 +
  <target name="assign1.build.call">
 +
    <!-- Call the target that does everything -->
 +
    <ant antfile="build.xml" target="assign1.test"/>
 +
  </target>
 +
  <target name="myTarget.check" depends="assign1.build.call">
 +
    <echo>The assign1.build was called!</echo>
 +
  </target>
 +
</project>
 +
</source>
 +
*2- '''Bash''' My notes on -> [[User:Minooz/RepoSyncProj/Bash | Min Bash]]
  
*'''Anthill'''
+
==Challenges==
 +
*1- I was receiving an error message while trying to do a new build on Hudson. It didn't let to clone the project on Hudson workspace. The error message is "Access is denied". So I had to delete the project and create a new one. Apparently one of the reasons is that Hudson doesn't let you queue the jobs. So, if you interrupt a job that is scheduled, you'll get the error message.
 +
*2- To run a bash script to build the project, there are 2 ways:
 +
:Execute Windows Batch Command:   
 +
<source lang=bash>
 +
D:\cygwin\bin\bash /home/HudsonPrac/buildHudson.sh
 +
</source>
 +
:Invoke an Ant Script
 +
<source lang=java>
 +
<project name="assign1" basedir=".">
 +
<target name="myTarget" >
 +
<exec executable="D:\cygwin\bin\bash" newenvironment="false">
 +
<arg value="/home/HudsonPrac/buildHudson.sh"/>
 +
</exec>
 +
</target>
 +
</project>
 +
</source>
 +
*3- Since the bash script is running in Windows platform through Cygwin, there are some conflicts,
 +
so the PATH was changed to: PATH=/usr/bin:$PATH
 +
Read more about some conflicts [http://cygwin.com/cygwin-ug-net/using.html#using-pathnames here]
 +
*4- If there was an error applying a patch to a repository e.g. <code> hg import patch12.patch </code>, there might be a problem with line numbers merging issue. So, there will be an error created with this message: "Hunk #1 Failed". There are some ways to remove the hunks manually or just ignore that patch, if the later versions are fixed.
 +
: By "hunk" I mean a "snippet of change", i.e. a part of the "diff". TortoiseHg uses this terminology and so does darcs. – Deniz Dogan
 +
: For Hunk Failed message, first we need to make sure it's not applied before. Second thing to check is that bases are the same. All the revisions before that new changeset even with failed build should be applied before applying the successfully built change-set.
 +
*5- If there is an uncommited message, the script can not work properly, e.g. Rev is 18+. So, error should be displayed for the uncommited change-sets.
 +
*6- To run JUnit Tests from command line:
 +
: add the JUnit installation directory and the junit.jar file and also QTjava to the CLASSPATH. e.g. <code>CLASSPATH=".;D:\cygwin\home\java\junit3.0.1\junit.jar;C:\Program Files\java\jre6\lib\ext\QTjava"</code>
 +
: to run a sample AllTests class go to the installation directory and run: <code> java junit.textui.TestRunner junit.samples.AllTests </code>
 +
: to run NexJ AllTests do:
 +
:: build the model: go to ws/core/build/  run <code> ant </code> or <code> ant -f build_JUnitTest.xml</code>
 +
:: from command line: go to ws/out/core/ run <code> java junit.textui.TestRunner nexj.core.AllTests</code> (getting error IOExcpetion) so we just ran one test file <code> java junit.textui.TestRunner nexj.core.util.MathUtilTest</code> (We did some changes to <code> core/test/nexj/core/util</code> to practice with it) Or we can run the ant target for test: <code> ant -f build_JUnitTest.xml test </code>
 +
*7- Bugs after implementation at NexJ
  
*'''Buildbot'''
+
: Working with remote repository; getting tip and log from remote repository is not as simple as moving to local repo and get all the information we need.  
:The [http://buildbot.net/trac Buildbot] is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure.[http://buildbot.net/buildbot/docs/current/Introduction.html#Introduction]
 
: [http://buildbot.net/buildbot/docs/current/index.html#Top Here] is the documentation for Buildbot. [https://developer.mozilla.org/presentations/buildbot-2.01.07.mov This video]also explains different features of this system.
 
:The buildmaster is usually fed Changes by some sort of version control system, which may cause builds to be run. As the builds are performed, various status messages are produced, which are then sent to any registered Status Targets.[[Image:Buildbot.png|thumb|none|widthpx|]]
 
  
:-Change Sources, which create a Change object each time something is modified in the VC repository. Most ChangeSources listen for messages from a hook script of some sort. Some sources actively poll the repository on a regular basis. All Changes are fed to the Schedulers.
+
: NexJ Internal repo has several branches, and when cloning and pulling, we just needed to mention it as a default branch, to not to confuse revision numbers and latest changesets.
:- Schedulers, which decide when builds should be performed. They collect Changes into BuildRequests, which are then queued for delivery to Builders until a buildslave is available.
+
: Since the decimal part of the revision number will keep changing from computer to computer, it was preferred to use the global hex chagesetID (GUID).
:-Builders, which control exactly how each build is performed (with a series of BuildSteps, configured in a BuildFactory). Each Build is run on a single buildslave.
+
: So, the first big change was adding this command to get the list of latest changesets up to the tip:
:-Status plugins, which deliver information about the build results through protocols like HTTP, mail, and IRC.
+
::<code>hg log ${IntDir} -r ${PrevRev}: -b default --template '{node}\n'</code>
*Common Requirements
+
: This number will also be added to the commit summary while importing the patch from Internal repo
:At a bare minimum, you'll need the following for both the buildmaster and a buildslave:
 
:-Python: http://www.python.org Buildbot requires python-2.4 or later.
 
:-Twisted: http://twistedmatrix.com (Twisted is an event-driven networking engine written in Python and licensed under the MIT license.)
 
:Requirements for Buildmaster
 
:-sqlite3: http://pypi.python.org/pypi/pysqlite
 
::The sqlite3 package is required for python-2.5 and earlier (it is already included in python-2.5 and later, but the version in python-2.5 has nasty bugs)
 
:-simplejson: http://pypi.python.org/pypi/simplejson
 
::The simplejson package is required for python-2.5 and earlier (it is already included as json in python-2.6 and later)
 
:-Jinja2: http://jinja.pocoo.org/2
 
::Jinja2 is a general purpose templating language and is used by Buildbot to generate the HTML output.
 
==To Learn==
 
: [[User:Minooz/RepoSyncProj/Ant | Min Ant]]
 
: [[User:Minooz/Mercurial | Min Mercurial]]
 

Latest revision as of 12:45, 17 November 2010

Repositories' Syncing

Version Control Tool

  1. Mercurial My notes on -> Min Mercurial
  2. CVS
  3. SVN
  4. Git

Continuous Integration System

  1. Hudson My notes on -> Min Hudson
  2. Cruise Control My notes on -> Min Cruise Control
  3. Buildbot My notes on -> Min Buildbot
  4. Apache Continuum
  5. Maven
  6. Anthill

Scripting

A new ant script is created e.g. buildHudson.xml that triggers the target(assign1.test) of main build file(build.xml) of the project. See below:
<project name="assign1" basedir="." default="myTarget">
  <target name="assign1.build.call">
    <!-- Call the target that does everything -->
    <ant antfile="build.xml" target="assign1.test"/>
   </target>
  <target name="myTarget.check" depends="assign1.build.call">
    <echo>The assign1.build was called!</echo>
  </target>
</project>

Challenges

  • 1- I was receiving an error message while trying to do a new build on Hudson. It didn't let to clone the project on Hudson workspace. The error message is "Access is denied". So I had to delete the project and create a new one. Apparently one of the reasons is that Hudson doesn't let you queue the jobs. So, if you interrupt a job that is scheduled, you'll get the error message.
  • 2- To run a bash script to build the project, there are 2 ways:
Execute Windows Batch Command:
D:\cygwin\bin\bash /home/HudsonPrac/buildHudson.sh
Invoke an Ant Script
<project name="assign1" basedir=".">
	<target name="myTarget" >
		<exec executable="D:\cygwin\bin\bash" newenvironment="false">
		<arg value="/home/HudsonPrac/buildHudson.sh"/>
		</exec>
	</target>
</project>
  • 3- Since the bash script is running in Windows platform through Cygwin, there are some conflicts,

so the PATH was changed to: PATH=/usr/bin:$PATH Read more about some conflicts here

  • 4- If there was an error applying a patch to a repository e.g. hg import patch12.patch , there might be a problem with line numbers merging issue. So, there will be an error created with this message: "Hunk #1 Failed". There are some ways to remove the hunks manually or just ignore that patch, if the later versions are fixed.
By "hunk" I mean a "snippet of change", i.e. a part of the "diff". TortoiseHg uses this terminology and so does darcs. – Deniz Dogan
For Hunk Failed message, first we need to make sure it's not applied before. Second thing to check is that bases are the same. All the revisions before that new changeset even with failed build should be applied before applying the successfully built change-set.
  • 5- If there is an uncommited message, the script can not work properly, e.g. Rev is 18+. So, error should be displayed for the uncommited change-sets.
  • 6- To run JUnit Tests from command line:
add the JUnit installation directory and the junit.jar file and also QTjava to the CLASSPATH. e.g. CLASSPATH=".;D:\cygwin\home\java\junit3.0.1\junit.jar;C:\Program Files\java\jre6\lib\ext\QTjava"
to run a sample AllTests class go to the installation directory and run: java junit.textui.TestRunner junit.samples.AllTests
to run NexJ AllTests do:
build the model: go to ws/core/build/ run ant or ant -f build_JUnitTest.xml
from command line: go to ws/out/core/ run java junit.textui.TestRunner nexj.core.AllTests (getting error IOExcpetion) so we just ran one test file java junit.textui.TestRunner nexj.core.util.MathUtilTest (We did some changes to core/test/nexj/core/util to practice with it) Or we can run the ant target for test: ant -f build_JUnitTest.xml test
  • 7- Bugs after implementation at NexJ
Working with remote repository; getting tip and log from remote repository is not as simple as moving to local repo and get all the information we need.
NexJ Internal repo has several branches, and when cloning and pulling, we just needed to mention it as a default branch, to not to confuse revision numbers and latest changesets.
Since the decimal part of the revision number will keep changing from computer to computer, it was preferred to use the global hex chagesetID (GUID).
So, the first big change was adding this command to get the list of latest changesets up to the tip:
hg log ${IntDir} -r ${PrevRev}: -b default --template '{node}\n'
This number will also be added to the commit summary while importing the patch from Internal repo