Difference between revisions of "How the Build Works"

From CDOT Wiki
Jump to: navigation, search
 
Line 17: Line 17:
 
<tr><th>Time</th><th>Topic</th><th>Details</th><tr>
 
<tr><th>Time</th><th>Topic</th><th>Details</th><tr>
  
<tr><td>00:00:00</td><td>Introduction</td><td>Professor David Humphrey introduces Mike Beltzner of the Mozilla Corporation and briefly touches on how Seneca College connected with the project.</td></tr>
+
<tr><td>00:00:13</td><td>Introducing J. Paul Reed</td><td>Professor David Humphrey of Seneca College introduces J. Paul Reed, Build Engineer, Mozilla Corporation.</td></tr>
  
<tr><td>00:02:15</td><td>The Mozilla Corporate Community</td><td>7 people in Canada: 3 in Ottawa, 3 in downtown Toronto, couple north of Toronto, some in Vancouver, 40 or 50 in California, Europe and Japan.</td></tr>
+
<tr><td>00:02:00</td><td>Lecture Agenda</td><td>
 +
Overview of the history of the build system.</td></tr>
  
<tr><td>00:03:04</td><td>Talk Agenda</td><td>What is the community, its place in the value chain, etiquette, points of entry, demos of entry.</td></tr>
+
<tr><td>00:03:00</td><td>"You Have the Wrong Guy"</td><td>Paul that he releases the software and considers himself more of a release engineer. You should be contacting the real build engineers Benjamin Smedberg, Mark Mentovai, Chris Seawood, and Bryan Ryner.</td></tr>
  
<tr><td>00:04:12</td><td>What the Community is</td><td>Everybody who is at all involved with the products (including code, discussion, testing, bug filing, submitting graphics, promoting the use).</td></tr>
+
<tr><td>00:03:47</td><td>Paul's Background</td><td>Paul sheds some light on his background with the Netscape Build team (1998), his work on VM's at VMWare as release engineer and how he came full circle back to the project as a Moco (Mozilla Corporation) employee.</td></tr>
  
<tr><td>00:04:46</td><td>Evangelizing Firefox</td><td>Promoting the use of the product with passionFirefox Crop Circle.</td></tr>
+
<tr><td>00:04:11</td><td>History of the build system</td><td>Paul mentions how Netscape had a collection of make files. Supporting 20 Unix platforms, Win16 & Win32.  The process was very mystical and brittleDocumentation on the process was very crude.</td></tr>
  
<tr><td>00:05:50</td><td>The Community Owns the Code</td><td>Unlike most companies, The Mozilla Project is totally open source.  The code isn't the asset.  Mike discusses this idea and the context in which the project works.</td></tr>
+
<tr><td>00:05:50</td><td>Code Rush Documentary</td><td>Paul mentions the PBS documentary on the challenges of having three outsiders build Mozilla on the 3 platforms (Windows, Unix and Macintosh).</td></tr>
  
<tr><td>00:06:32</td><td>Value Chain for Software</td><td>How can people contribute and add value to users? It's everywhere!  Who uses Mozilla products, how do they give back and what is their position in the value chain.</td></tr>
+
<tr><td>00:06:58</td><td>Autotools</td><td>On the limited use of autoconf and lack of usage of automake and other utilities. Paul also elaborates on the autoconf systems usage of .in files and how they drive/fit into the build process.</td></tr>
  
<tr><td>00:09:25</td><td>Mozilla = Community</td><td>No one acts independently of the community -- "we live and die by the community", Mike Shaver.</td></tr>
+
<tr><td>00:07:50</td><td>No Autoconf?</td><td>Paul explains Why don't we need autoconf to build Mozilla?  An old machine at Mozilla monitors code checked (specifically config.in which is the source config file) in an automatically do this.</td></tr>
  
<tr><td>00:10:00</td><td>Roles of the Players</td><td>Mozilla Foundation (guidance), Mozilla Corporation (thrust), Community (passion, innovation, reach).</td></tr>
+
<tr><td>00:08:46</td><td>Major Organs of the build system Explained.</td><td>Paul discusses Netscape Portable Runtime (NSPR), Netscape Security Service (NSS), C-LDAP SDK, JavaScript, Supporting Libraries and most importantly the actual Mozilla products (Firefox, Thunderbird, Sunbird etc.) All the above are in top level citizens, you get them whether or not you want them when you check out the code.</td></tr>
  
<tr><td>00:12:41</td><td>Mozilla is a Community That Happens to Make Software</td><td>Mozilla makes it possible for the community to work together around a single "thing". This idea is not limited to software; to collaborate and produce something as a group could apply beyond software.</td></tr>
+
<tr><td>00:10:24</td><td>client.mk - Useful targets</td><td>
 +
* '''checkout''' - grabs checkout of the source code (done this way so they can version the different modules - a feature that cvs does not support.)
 +
* '''build''' - builds the application
 +
* '''configure''' - run automatically when build is run
 +
* '''fast-update''' - checks Bonsai and does an update and retrieves only the parts you are working on (unlike cvs update).</td></tr>
  
<tr><td>00:13:50</td><td>Netiquette</td><td>How to deal with people online after losing the social context of in-person communication. Some basic rules with interacting in an online community for your success.</td></tr>
+
<tr><td>00:13:07</td><td>.mozconfig</td><td>Paul explains the importance of the .mozconfig file in the build process, and mk_add_option vs ac_add_options.
 +
Build Configurator is here to help (http://webtools.mozilla.org/build/config.cgi)</td></tr>
  
<tr><td>00:18:03</td><td>Meritocracy of Open Source</td><td>Mike describes the difference between classical power structures and open source power structuresDeeds speak in Open Source. You're always being judged and you must strive to build social capital (aka whuffie).</td></tr>
+
<tr><td>00:15:43</td><td>Interesting things about build configuration.</td><td>
 +
The system does not guessIf the compile fails it knows it could not find it.
 +
If the compiler knows where to find it and build system does not its still OK.</td></tr>
  
<tr><td>00:21:15</td><td>Points of Entry</td><td>From the consumer to the developer--using products, weblogs, web forums, newsgroups, IRC chat, status calls, BugZilla and CVS.</td></tr>
+
<tr><td>00:19:00</td><td>10 Lines</td><td>The 10 most important lines in build process.  The entire process for browser, Thunderbird etc. is driven by these 10 lines</td></tr>
  
<tr><td>00:22:55</td><td>All Points of Entry Intersect</td><td>Design, testing, and advocacy tasks, for example, are coordinated by and utilize all points of entry.</td></tr>
+
<tr><td>00:19:54</td><td>Explanation of the Tiers</td><td>Paul discusses the various tiers in the build system and their contents.</td></tr>
  
<tr><td>00:26:04</td><td>IN-PRODUCT: Report a Broken Website</td><td>A website that doesn't render properly can be reported using the in-browser reporting toolThis provides a low-barrier to entry to participate in the Mozilla community.</td></tr>
+
<tr><td>00:23:22</td><td>Two Pass Build System Explained.</td><td>Paul elaborates on the export and libs pass (and sometimes also a thrid pass called deps). The  Export pass copies all the .h files for all modules. The libs pass creates libraries that are needed for other modules. The Deps pass handles the automatic dependencies for files that you have edited etcHe stressed that all of this is still driven by the rules.mk file.</td></tr>
  
<tr><td>00:27:02</td><td>IN-PRODUCT: Report a Web Forgery</td><td>In-browser phishing site reporting tool goes directly into the fishing filter to benefit the community again, instantly warning other users who stumble across the site.</td></tr>
+
<tr><td>00:25:00</td><td>Explanation of the rules.mk file.</td><td>Good news: You need not necessarily worry about it.  Just know what the definitions are if you need to use them.  Paul also explains that the reason the makefile so short it is driven by the rules.mk file.</td></tr>
  
<tr><td>00:27:48</td><td>IN-PRODUCT: Help System Links to Websites</td><td>Whenever you query the help sites they are able to know what you are looking forIn this way, just by looking for help you can contribute to The Mozilla Project.</td></tr>
+
<tr><td>00:28:09</td><td>More rules.mk variables</td><td>Paul explains you can provide the build system will the all the information in needs to create your files.  Chrome is stored in .jar filesThe build system can handle .jars on their own, there need not be any rules. EXTRA_PP_COMPONENTS similar to C pre-processor defines in C.</td></tr>
  
<tr><td>00:28:20</td><td>WEBLOG: http://planet.mozilla.org/</td><td>Weblog planet is an aggregation website that gathers together the weblogs of prominent members in the community to keep people abreast of what other people are doing.  This is used to get a sense of what people are working on; at Mozilla there are no status reports, so blogging lets them know what's going on.</td></tr>
+
<tr><td>00:30:47</td><td>MDC</td><td>Paul mentions that MDC has some great documentation on build flags and other build documentation.</td></tr>
  
<tr><td>00:30:46</td><td>WEBFORUMS: http://www.mozillazine.org/</td><td>The front page is a summary of the high-level news item in the Mozilla world. This website is owned by the corporation or foundation people. MozillaZine acts as a support system for Mozilla as the corporation does not have any support staff employed of its own.</td></tr>
+
<tr><td>00:33:43</td><td>Local vs. Official Builds</td><td>Paul explains the difference between a local build and the official ones from Mozilla, i.e. -d build ID (datestring), post build packaging commencing - for official builds it uploads to ftp mirrors (unlike local builds which just allow you to run them), talk back reports sent back to Mozilla.</td></tr>
  
<tr><td>00:33:48</td><td>WEBFORUMS: http://spreadfirefox.com</td><td>Spreadfirefox.com is a set of forums and weblogs that are all about spreading Firefox. The people involved are not necessarily people that work on the code. The goal is to just get people to use Firefox. Participants are rewarded with points and "ranked".</td></tr>
+
<tr><td>00:35:36</td><td>Interesting parts of the Mozilla Build system.</td><td>Paul explains how the build system "cruft" and why it seems "weird". He discusses the reasons behind this. Specifically the use of autoconf as a thin wrapper as well as the issues accompanying use of top level directories in CVS.</td></tr>
  
<tr><td>00:35:20</td><td>WIKIS</td><td>
+
<tr><td>00:38:23</td><td>The system is not as self-contained as it should be.</td><td>Currently localization packaging updates and generation are part of Tinderbox right now but this is not where it should beThe right place for them would be the build system.</td></tr>
Wiki.mozilla.org is the main Wiki for The Mozilla Project.  All project-related plans and in-progress code documentation will first exist in this WikiEventually, this information will be migrated to the Mozilla Development Centre.
 
  
* Mozilla Developer Centre - http://developer.mozilla.org (devmo) is the Wiki used to provide documentation for people who are working on Mozilla code.
+
<tr><td>00:39:22</td><td>Re-inventing the wheel</td><td>Paul discusses why Mozilla has taken a not-invented here attitude. In essence he explains why they have re-written things already in existence on most modern systems.</td></tr>
* DevNews Blog - http://developer.mozilla.org/devnews/ carries announcements of latest releases and plans.
 
* WebWatch - http://developer.mozilla.org/webwatch/ announces cool things happening in the web community.</td></tr>
 
  
<tr><td>00:39:53</td><td>Newsgroups</td><td>Newsgroups are used by Mozilla for design discussions of the code, planning discussions about the release of code, and managing Mozilla subcommunities. dev.planning dev.general dev.apps.firefox are the busiest newsgroups.</td></tr>
+
<tr><td>00:40:40</td><td>Cross Platform Development</td><td>Paul discusses the remarkable fact that the browser can run on all three platforms without limited use of #ifdef pre-processor statements.  He remarked on the use of the NSPR and its benefits. Other oddities discussed include why C++ exceptions,templates and the 33 other portability rules (http://www.mozilla.org/hacking/portable-cpp.html#portability_rules)</td></tr>
  
<tr><td>00:41:48</td><td>Time Zones</td><td>Whenever you talk about a time, communicate the time zone because you are dealing with the world community.</td></tr>
+
<tr><td>00:44:50</td><td>Futures of the Build system</td><td>Paul discusses where the build system may be headed: Scons, Componentized builds, Moving to SVN</td></tr>
  
<tr><td>00:42:47</td><td>IRC chat</td><td>IRC is the place where the most active community members hang out.  This is the "pulse of what's happening at the moment" according to Mike, and the "quickest way to get up to speed in the community".</td></tr>
+
<tr><td>00:48:18</td><td>Re-write the whole thing.</td><td>Benjamin Smedberg threatens to re-write large (if not the whole) build system as the build engineers are starting to agree it may be easier to do that then to add stuff to it.</td></tr>
  
<tr><td>00:50:50</td><td>Search Before Asking Questions</td><td>Try to find answers to your own questions before asking others--this is a better use of their time.</td></tr>
+
<tr><td>00:48:38</td><td>Useful notes for new developers</td><td>-DDEBUG-username useful for checking specific stuff when you yourself are debugging (can be turned off when you are done debugging).  --No remote -p prevents your existing Firefox profile from being corrupted when you build a custom version.</td></tr>
  
<tr><td>00:58:00</td><td>Status Calls</td><td>Status of a project over a 1-800 number. This provides a low bar to entry, yet remains very detailed and is an "active" activity with low signal-to-noise ratio.</td></tr>
+
<tr><td>00:50:16</td><td>Helpful Tools</td><td>Paul discussed the use of other tools developers use and which would be useful to new entrants to the Firefox community. For example, MDC, tinderbox, LXR, Planet etc.</td></tr>
  
<tr><td>00:58:55</td><td>Bugzilla</td><td>Bugzilla is essentially a tool where everything happens.  Bug entries include a description of a problem, steps to reproduce, actual results that people see, and expected results.  People can reply to the bug and add comments.  Someone eventually takes the bug and owns it and drives it to completion--eventually resolving the issue.</td></tr>
+
<tr><td>00:51:06</td><td>Obtaining Help</td><td>Paul explains that #build is meant for official builds and to talk to build engineers. They like to keep a high signal to roise For general questions and inquiries use #developers or #Seneca. However, he stressed that it is important not to hesitate to ask for help.</td></tr>
 
 
<tr><td>01:02:20</td><td>How to File a Bug</td><td>Filing a bug in Bugzilla.</td></tr>
 
 
 
<tr><td>01:05:24</td><td>Tinderbox Page</td><td>The Tinderbox page is a communication vehicle showing the status of the latest build machines that are building a set of code.  It also shows blocking bugs (bugs which currently block a new release).</td></tr>
 
 
 
<tr><td>01:06:58</td><td>Bonsai and CVS</td><td>Bonsai allows people to track CVS activity on the code base, showing the last check-ins to the code tree.</td></tr>
 
 
 
<tr><td>01:08:41</td><td>Which Point of Entry is Appropriate for a Task</td><td>&nbsp;</td></tr>
 
 
</table>
 
</table>

Revision as of 15:21, 18 January 2007

Talk Details

  • Date: Sept 20, 2006
  • Speaker: J. Paul Reed

Media

Talk Outline

TimeTopicDetails
00:00:13Introducing J. Paul ReedProfessor David Humphrey of Seneca College introduces J. Paul Reed, Build Engineer, Mozilla Corporation.
00:02:00Lecture Agenda Overview of the history of the build system.
00:03:00"You Have the Wrong Guy"Paul that he releases the software and considers himself more of a release engineer. You should be contacting the real build engineers Benjamin Smedberg, Mark Mentovai, Chris Seawood, and Bryan Ryner.
00:03:47Paul's BackgroundPaul sheds some light on his background with the Netscape Build team (1998), his work on VM's at VMWare as release engineer and how he came full circle back to the project as a Moco (Mozilla Corporation) employee.
00:04:11History of the build systemPaul mentions how Netscape had a collection of make files. Supporting 20 Unix platforms, Win16 & Win32. The process was very mystical and brittle. Documentation on the process was very crude.
00:05:50Code Rush DocumentaryPaul mentions the PBS documentary on the challenges of having three outsiders build Mozilla on the 3 platforms (Windows, Unix and Macintosh).
00:06:58AutotoolsOn the limited use of autoconf and lack of usage of automake and other utilities. Paul also elaborates on the autoconf systems usage of .in files and how they drive/fit into the build process.
00:07:50No Autoconf?Paul explains Why don't we need autoconf to build Mozilla? An old machine at Mozilla monitors code checked (specifically config.in which is the source config file) in an automatically do this.
00:08:46Major Organs of the build system Explained.Paul discusses Netscape Portable Runtime (NSPR), Netscape Security Service (NSS), C-LDAP SDK, JavaScript, Supporting Libraries and most importantly the actual Mozilla products (Firefox, Thunderbird, Sunbird etc.) All the above are in top level citizens, you get them whether or not you want them when you check out the code.
00:10:24client.mk - Useful targets
  • checkout - grabs checkout of the source code (done this way so they can version the different modules - a feature that cvs does not support.)
  • build - builds the application
  • configure - run automatically when build is run
  • fast-update - checks Bonsai and does an update and retrieves only the parts you are working on (unlike cvs update).
00:13:07.mozconfigPaul explains the importance of the .mozconfig file in the build process, and mk_add_option vs ac_add_options. Build Configurator is here to help (http://webtools.mozilla.org/build/config.cgi)
00:15:43Interesting things about build configuration.

The system does not guess. If the compile fails it knows it could not find it.

If the compiler knows where to find it and build system does not its still OK.
00:19:0010 LinesThe 10 most important lines in build process. The entire process for browser, Thunderbird etc. is driven by these 10 lines
00:19:54Explanation of the TiersPaul discusses the various tiers in the build system and their contents.
00:23:22Two Pass Build System Explained.Paul elaborates on the export and libs pass (and sometimes also a thrid pass called deps). The Export pass copies all the .h files for all modules. The libs pass creates libraries that are needed for other modules. The Deps pass handles the automatic dependencies for files that you have edited etc. He stressed that all of this is still driven by the rules.mk file.
00:25:00Explanation of the rules.mk file.Good news: You need not necessarily worry about it. Just know what the definitions are if you need to use them. Paul also explains that the reason the makefile so short it is driven by the rules.mk file.
00:28:09More rules.mk variablesPaul explains you can provide the build system will the all the information in needs to create your files. Chrome is stored in .jar files. The build system can handle .jars on their own, there need not be any rules. EXTRA_PP_COMPONENTS similar to C pre-processor defines in C.
00:30:47MDCPaul mentions that MDC has some great documentation on build flags and other build documentation.
00:33:43Local vs. Official BuildsPaul explains the difference between a local build and the official ones from Mozilla, i.e. -d build ID (datestring), post build packaging commencing - for official builds it uploads to ftp mirrors (unlike local builds which just allow you to run them), talk back reports sent back to Mozilla.
00:35:36Interesting parts of the Mozilla Build system.Paul explains how the build system "cruft" and why it seems "weird". He discusses the reasons behind this. Specifically the use of autoconf as a thin wrapper as well as the issues accompanying use of top level directories in CVS.
00:38:23The system is not as self-contained as it should be.Currently localization packaging updates and generation are part of Tinderbox right now but this is not where it should be. The right place for them would be the build system.
00:39:22Re-inventing the wheelPaul discusses why Mozilla has taken a not-invented here attitude. In essence he explains why they have re-written things already in existence on most modern systems.
00:40:40Cross Platform DevelopmentPaul discusses the remarkable fact that the browser can run on all three platforms without limited use of #ifdef pre-processor statements. He remarked on the use of the NSPR and its benefits. Other oddities discussed include why C++ exceptions,templates and the 33 other portability rules (http://www.mozilla.org/hacking/portable-cpp.html#portability_rules)
00:44:50Futures of the Build systemPaul discusses where the build system may be headed: Scons, Componentized builds, Moving to SVN
00:48:18Re-write the whole thing.Benjamin Smedberg threatens to re-write large (if not the whole) build system as the build engineers are starting to agree it may be easier to do that then to add stuff to it.
00:48:38Useful notes for new developers-DDEBUG-username useful for checking specific stuff when you yourself are debugging (can be turned off when you are done debugging). --No remote -p prevents your existing Firefox profile from being corrupted when you build a custom version.
00:50:16Helpful ToolsPaul discussed the use of other tools developers use and which would be useful to new entrants to the Firefox community. For example, MDC, tinderbox, LXR, Planet etc.
00:51:06Obtaining HelpPaul explains that #build is meant for official builds and to talk to build engineers. They like to keep a high signal to roise For general questions and inquiries use #developers or #Seneca. However, he stressed that it is important not to hesitate to ask for help.