Open main menu

CDOT Wiki β

Difference between revisions of "Koji Follow"

(Run Notes)
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
= Koji Follow Que Script =
 
= Koji Follow Que Script =
  
== Notes ==
+
== Build Notes ==
  
 
* This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
 
* This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
Line 10: Line 10:
 
** latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order
 
** latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order
  
== Possible Breakage Scenarios ==
+
== Common Repository Breakage Scenarios due to basic circular dependencies ==
  
=== Scenario 1 ===
+
=== All of the following examples take place after this series of events ===
  
 +
* Start of initial-build
 
# libA-1.0-1.fc17 is built (Provides: libA.so.1)
 
# libA-1.0-1.fc17 is built (Provides: libA.so.1)
# packageB-1.0-1.fc17 depends on packageC but is bootstrapped without it and built
+
# packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
# packageC-1.0-1.fc17 depends on libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
+
# packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
# packageB-1.0-2.fc17 depends on packageC and is built with packageC-1.0-1.fc17
+
# packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
 +
 
 +
=== Building a new library first and the packages that depend on it later ===
 +
 
 +
* Start of mass-rebuild
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
## Attempting to build the full version of packageB-2.0-1.fc18 will depend on packageC-1.0-1.fc17 which requires libA.so.1 which only find libA.so.2 with the current repodata (this package needs to be bootstrapped again)
+
#* Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata
## Attempting to build packageC-2.0-1.fc18 will depend on packageB-1.0-2.fc17 which will depend on packageC-1.0-1.fc17 which brings us to the breakage error above
+
 
 +
=== Building new packages first and then the following libraries they depend on after ===
  
=== Scenario 2 ===
+
* Start of mass-rebuild
 +
# packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
 +
# packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
 +
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
 +
#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2
  
# libA-1.0-1.fc17 is built (Provides: libA.so.1)
+
== Run Notes ==
# packageB-1.0-1.fc17 depends on packageC but is bootstrapped without it and built
+
 
# packageC-1.0-1.fc17 depends on libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
+
* Packages are not processed in alphabetical order and the output is not displayed in numerical order
# packageB-1.0-2.fc17 depends on packageC and is built with packageC-1.0-1.fc17
+
* The koji error log is for processing messages and the koji stdout log is for queing messages
#
+
* You need to add any new package names to a tag before they are built
 +
* You need a lot of free space in the current directory that the script is running in
 +
* Before running the script, attempt a scratch build of a sample package with the given build tag
 +
* You can redirect stderr to stdout and write stdout to a file so that the messages are in order with each other
  
 
= Downloads =
 
= Downloads =
Line 66: Line 79:
 
</pre>
 
</pre>
  
<b>python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log</b>
+
<b>python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log</b>
  
 
== latest-follow mode ==
 
== latest-follow mode ==
Line 95: Line 108:
 
</pre>
 
</pre>
  
<b>python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log</b>
+
== Execution Example ==
 +
 
 +
<b>python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log</b>
  
= Example Output =
+
= Example Log Output =
  
 
== STDOUT Messages ==
 
== STDOUT Messages ==

Latest revision as of 10:27, 8 July 2013

Koji Follow Que Script

Build Notes

  • This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
  • This script does not automatically solve bootstrapping or library.so name bump requires issues (which happens often during a completely new tag rebuild)
  • This script does not attempt to solve any package build failure issues
  • This script can be run in 2 modes: "mass-rebuild" and "latest-follow" and ques builds in order based on their "creation_ts" value
    • mass-rebuild mode will attempt to compile a complete list of packages that were built for a given tag and then que them in the given order
    • latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order

Common Repository Breakage Scenarios due to basic circular dependencies

All of the following examples take place after this series of events

  • Start of initial-build
  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
  3. packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
  4. packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)

Building a new library first and the packages that depend on it later

  • Start of mass-rebuild
  1. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata

Building new packages first and then the following libraries they depend on after

  • Start of mass-rebuild
  1. packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
  2. packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
  3. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2

Run Notes

  • Packages are not processed in alphabetical order and the output is not displayed in numerical order
  • The koji error log is for processing messages and the koji stdout log is for queing messages
  • You need to add any new package names to a tag before they are built
  • You need a lot of free space in the current directory that the script is running in
  • Before running the script, attempt a scratch build of a sample package with the given build tag
  • You can redirect stderr to stdout and write stdout to a file so that the messages are in order with each other

Downloads

Example Config and Execution Scenarios

mass-rebuild mode

~/koji-follow.conf

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "all"
retry_build = "all"

python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log

latest-follow mode

~/koji-follow.conf

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "latest"
retry_build = "failed"

Execution Example

python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log

Example Log Output

STDOUT Messages

[info] pkg_process: [1/6604]: {'owner_name': 'jkeating', 'package_name': 'email2trac', 'task_id': 1503441, 'creation_event_id': 1793404, 'creation_time': '2009-07-25 14:59:59.646649', 'epoch': None, 'nvr': 'email2trac-0.13-6.fc12', 'name': 'email2trac', 'completion_time': '2009-07-25 15:35:37.427707', 'state': 1, 'version': '0.13', 'package_id': 6215, 'volume_name': 'DEFAULT', 'release': '6.fc12', 'creation_ts': 1248533999.64665, 'completion_ts': 1248536137.42771, 'id': 117394, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [email2trac-0.13-6.fc12.src.rpm] -> [cli-build/1362515245.29142.RuqZKocF]
[info] pkg_process: [2/6604]: {'owner_name': 'jkeating', 'package_name': 'powerpc-utils-papr', 'task_id': 1525540, 'creation_event_id': 1839140, 'creation_time': '2009-07-28 04:43:55.459368', 'epoch': None, 'nvr': 'powerpc-utils-papr-1.1.6-2.fc12', 'name': 'powerpc-utils-papr', 'completion_time': '2009-07-28 05:47:09.877729', 'state': 1, 'version': '1.1.6', 'package_id': 5369, 'volume_name': 'DEFAULT', 'release': '2.fc12', 'creation_ts': 1248756235.45937, 'completion_ts': 1248760029.87773, 'id': 122145, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [powerpc-utils-papr-1.1.6-2.fc12.src.rpm] -> [cli-build/1362515248.044621.imzInFfA]
[info] pkg_process: [3/6604]: {'owner_name': 'jkeating', 'package_name': 'ppc64-utils', 'task_id': 1525552, 'creation_event_id': 1839195, 'creation_time': '2009-07-28 04:46:54.025335', 'epoch': None, 'nvr': 'ppc64-utils-0.14-5.fc12', 'name': 'ppc64-utils', 'completion_time': '2009-07-28 05:51:18.860184', 'state': 1, 'version': '0.14', 'package_id': 131, 'volume_name': 'DEFAULT', 'release': '5.fc12', 'creation_ts': 1248756414.02534, 'completion_ts': 1248760278.86018, 'id': 122152, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [ppc64-utils-0.14-5.fc12.src.rpm] -> [cli-build/1362515249.622192.vcbTPtpj]
[info] pkg_process: [4/6604]: {'owner_name': 'rrakus', 'package_name': 'iprutils', 'task_id': 1787705, 'creation_event_id': 2291747, 'creation_time': '2009-11-04 05:55:36.885871', 'epoch': None, 'nvr': 'iprutils-2.2.18-1.fc12', 'name': 'iprutils', 'completion_time': '2009-11-04 05:58:25.827132', 'state': 1, 'version': '2.2.18', 'package_id': 722, 'volume_name': 'DEFAULT', 'release': '1.fc12', 'creation_ts': 1257314136.88587, 'completion_ts': 1257314305.82713, 'id': 139717, 'volume_id': 0, 'owner_id': 613}
        [info] que_build: [iprutils-2.2.18-1.fc12.src.rpm] -> [cli-build/1362515252.116506.kQNgDlWL]
[info] pkg_process: [5/6604]: {'owner_name': 'sharkcz', 'package_name': 'gtksourceview2-sharp', 'task_id': 3035160, 'creation_event_id': 3682022, 'creation_time': '2011-04-29 07:07:04.359159', 'epoch': None, 'nvr': 'gtksourceview2-sharp-1.0-10.svn89788.fc16', 'name': 'gtksourceview2-sharp', 'completion_time': '2011-04-29 07:09:55.860531', 'state': 1, 'version': '1.0', 'package_id': 6142, 'volume_name': 'DEFAULT', 'release': '10.svn89788.fc16', 'creation_ts': 1304060824.35916, 'completion_ts': 1304060995.86053, 'id': 241357, 'volume_id': 0, 'owner_id': 244}
        [info] que_build: [gtksourceview2-sharp-1.0-10.svn89788.fc16.src.rpm] -> [cli-build/1362515254.528944.WTNbsaqJ]
...

STDERR Messages

[info] primary_tag: [1188/1188/1584]: rwho
[info] primary_tag: [396/396/792]: jlatexmath
[info] primary_tag: [0/0/396]: fprintd
[info] primary_tag: [1584/1584/1980]: objectweb-asm4
[info] primary_tag: [1980/1980/2376]: ktp-approver
[info] primary_tag: [792/792/1188]: nginx
[info] primary_tag: [3168/3168/3564]: audex
[info] primary_tag: [2376/2376/2772]: python-sh
[info] primary_tag: [3564/3564/3960]: python-cmd2
[info] primary_tag: [2772/2772/3168]: perl-IPC-Run-SafeHandles
...
        [info] file_delete: aplus-fsf-4.22.4-21.fc16.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/aplus-fsf/4.22.4/21.fc16/src/aplus-fsf-4.22.4-21.fc16.src.rpm] -> [aplus-fsf-4.22.4-21.fc16.src.rpm]
        [info] file_delete: tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/tritonus/0.3.7/0.13.20101108cvs.fc17/src/tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm] -> [tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm]
        [info] file_delete: jffi-1.0.10-1.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/jffi/1.0.10/1.fc17/src/jffi-1.0.10-1.fc17.src.rpm] -> [jffi-1.0.10-1.fc17.src.rpm]
        [info] file_delete: nautilus-sendto-trac-0.4-2.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/nautilus-sendto-trac/0.4/2.fc17/src/nautilus-sendto-trac-0.4-2.fc17.src.rpm] -> [nautilus-sendto-trac-0.4-2.fc17.src.rpm]
        [info] file_delete: sweep-0.9.3-9.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/sweep/0.9.3/9.fc17/src/sweep-0.9.3-9.fc17.src.rpm] -> [sweep-0.9.3-9.fc17.src.rpm]
...