From Symbian Developer Community

This article explains how to install the open source Hudson continuous integration tool, and how to configure and use it to perform Symbian platform and package builds.

Contents 1 Introduction 2 Installation 3 Configuration 3.1 Mercurial Plugin 3.2 Download Updates 4 Projects 4.1 PDK_Helium 4.2 PDK_Raptor 4.3 FBF_PrepareEnvironment 4.4 FBF_BuildPackage 5 Building 5.1 Pre-requisites 5.2 Start Build 5.3 Success or Fail 6 What next? 7 Open Defects 8 Further Info

Introduction

Hudson is the front-end tool that the Symbian Foundation use to build the platform and kits. We recommend that 3rd party developers also use Hudson because it is easier and more repeatable than the manual build process, and provides a centralised place for building the platform with Helium/FBF or SBSv2 ("Raptor"), and individual packages with Helium.

This article first explains how to install and configure Hudson for builds; we've made the process very simple by providing a Hudson "job" that you can install to download the build jobs from Mercurial (the Symbian Foundation configuration management tool). The article then outlines what other tools you'll need, and how you go about starting a build and interpreting the results.

Note that this article does not go into particular detail on how to use and interpret the results of each of the build jobs - instead each Hudson job configuration contains notes on how it is used and a pointer to the manual process it follows (this makes it much easier to keep the documentation accurate when we update the tools and add new build jobs).

Installation

Hudson is both quick and easy to install:

• Download hudson.war from http://hudson-ci.org/ and put it into its own working directory e.g C:\apps\Hudson
  • Note: v1.29 is the minimum version supported.
• Set HUDSON_HOME variable to the working directory, so that it knows where to unpack the content.

 set HUDSON_HOME=c:\apps\Hudson

• Ensure that EPOCROOT is set correctly

 set EPOCROOT=\

• Run the .war file

  java -jar hudson.war

The .war content will then be extracted into the HUDSON_HOME location. Console output will look something like -

 
C:\apps\Hudson>java -jar hudson.war
Running from: C:\apps\Hudson\hudson.war
[Winstone 2009/11/02 12:54:58] - Beginning extraction from war file
hudson home directory: c:\apps\Hudson
[Winstone 2009/11/02 12:55:01] - HTTP Listener started: port=8080
[Winstone 2009/11/02 12:55:01] - AJP13 Listener started: port=8009
[Winstone 2009/11/02 12:55:01] - Winstone Servlet Engine v0.9.10 running: controlPort=disabled
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\active-directory.hpi
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\crowd.hpi
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\maven-plugin.hpi
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\mercurial.hpi
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\scis-ad.hpi
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\ssh-slaves.hpi
02-Nov-2009 12:55:02 hudson.ClassicPluginStrategy createPluginWrapper
INFO: Loading plugin: c:\apps\Hudson\plugins\subversion.hpi
02-Nov-2009 12:55:03 hudson.model.Hudson load
INFO: Loading in 4 parallel threads
02-Nov-2009 12:55:06 hudson.model.Hudson load
INFO: Took 2422 ms to load
02-Nov-2009 12:55:06 hudson.TcpSlaveAgentListener <init>
INFO: JNLP slave agent listener started on TCP port 1773
 

• Now verify installation by going to http://localhost:8080. If the default port 8080 is in use, or you want to use another, then you can specify the required port when you launch Hudson by adding the following to the command line: --httpPort=port_number.

Congratulations, Hudson is now up and running.

Configuration

The configuration process adds a "Download Updates" job which we then use to get the jobs (from developer.symbian.org) that actually do platform/package building. First however we need to install the Mercurial Plugin for Hudson.

Mercurial Plugin

The plugin is installed through the Hudson web interface:

1. Select Manage Hudson->Manage Plugins->Available
2. Scroll down and tick the "Mercurial Plugin"
3. Hit the Install Button

The Manage Plugins page should show "Success" and console window output should contain:

INFO: JNLP slave agent listener started on TCP port 1662
02-Nov-2009 14:36:35 hudson.model.UpdateCenter$DownloadJob run
INFO: Starting the installation of Mercurial Plugin on behalf of anonymous
02-Nov-2009 14:36:36 hudson.model.UpdateCenter$UpdateCenterConfiguration download
INFO: Downloading Mercurial Plugin
02-Nov-2009 14:36:37 hudson.model.UpdateCenter$DownloadJob run
INFO: Installation successful: Mercurial Plugin

Shutdown Hudson by pressing ctrl-c in the console window -

 
[Winstone 2009/11/02 14:46:15] - Control thread shutdown successfully
[Winstone 2009/11/02 14:46:15] - Winstone shutdown successfully
 

Download Updates

Next we add the Download Updates job into Hudson. This job gets the configuration files for the jobs that are used to build the platform (from developer.symbian.org) and installs them into your Hudson installation. Note, this can be run again to keep the build jobs updated to the latest version.

To add the job you need to download File:Download Updates Config.xml, rename it to config.xml and copy it into a directory %HUDSON_HOME%\jobs\Download_Updates (you may need to create the directory):

 
mkdir %HUDSON_HOME%\jobs\Download_Updates
copy [[File:Download_Updates_Config.xml]] %HUDSON_HOME%\jobs\Download_Updates
cd  %HUDSON_HOME%\jobs\Download_Updates
ren Download_Updates_Config.xml config.xml
 

The "Download Updates" task should now be in the correct place and named properly. Now startup Hudson again:

java -jar hudson.war

You should now see that the "Download Updates" job is available on the Hudson webpage. This needs to be run to obtain the latest configs and is executed by selecting the icon on the right.

Selecting the "Download Updates" job link will then give a history list, a Blue icon indicates that the job was run successfully.

Once updates have been downloaded you should reload the configuration from disk. You can do this by either restarting Hudson - or by selecting the "Reload from Disk" option in the Hudson Management page. The latter is the preferable choice as it will not interupt any other running jobs.

Once you've reloaded the configurations from disk you should see the a number of new jobs appear on the dashboard (as shown below).

Projects

The default jobs (at time of writing) allow you to build the platform using either Helium/FBF (PDK_Helium) or SBSv2 (PDK_Raptor), and to build individual packages with Helium (FBF_BuildPackage), having first set up a build environment (FBF_PrepareEnvironment).

A little information is provided about each job below - you can find out more information about each job within its Hudson description page (select its hyperlink in the "job" column).

PDK_Helium

This project rebuilds the Symbian platform from source using Helium/Foundation Build Framework.

• It encapsulates the steps as defined in the manual build process for Helium.
• The source can be automatically downloaded using Mercurial or extracted from PDK zips (that you will need to manually download from the kits page).
• This build has been tested on Symbian^2 and Symbian^3.

PDK_Raptor

This project rebuilds the Symbian platform from source using SBSv2

• It encapsulates the steps as defined in the manual build process for Raptor and is a simple way to start building using SBSv2.
• The source can be automatically downloaded using Mercurial or extracted from PDK zips (that you will need to manually download from the kits page.)
• This build has been tested on Symbian^2 and Symbian^3.

FBF_PrepareEnvironment

This project prepares a build environment that can be used by the FBF_BuildPackage job for building packages (using the Foundation Build Framework (Helium)). It has been tested against the Symbian^2 and Symbian^3 PDK.

FBF_BuildPackage

This project builds a specified package against a build environment created using the FBF_PrepareEnvironment project (using the Foundation Build Framework (Helium)). The project can build the specified package from source already unzipped in the environment, or you can specify that you want the source cloned from Mercurial (this will be done at the correct revision as specified in the BOM (bill of materials, delivered with each release in build_BOM.zip).

Building

Pre-requisites

The set of tools needed for each build job is linked from its Hudson description page - nominally the set listed here: Tools Q&26As.

The archives/source needed for each build job are linked from its Hudson description page. These differ depending on what you are trying to build - for example you don't necessarily need source archives to build a package.

Building requires the location of a Physical disk on your machine with approx 50gb free for a single target (such as winscw_udeb). Building all 4 targets will require up to 120Gb of physical disc space.

Start Build

To build a job, select the Schedule a build link next to the job in the dashboard (as you did when downloading updates), or the Build Now side-bar option from within the job description page.

Start by selecting the "Build Now" option and providing the details into the form (form for Helium platform build is shown below). Note that when you start the build Hudson will first create a unique directory in the BUILD_DRIVE (named after the project and number of times the project has been run) and then subst it to the drive you specify in SUBST_DRIVE (unmapping any existing subst). If you want to build from source in PDK archives you'll need to set the source location (SRC_LOCATION) as Zipfiles and specify the RELEASE_ZIP_LOCATION where you've downloaded the required PDK archives.

The rest of the fields are self explanatory (or have an explanation).

Once you've provided details for the build to start, you can see its progress by selecting the bar on the left hand side. Selecting it will display the console output.

• When the job starts, mandatory zipfiles such as binaries_epoc_additional.zip are unpacked. Output should be similar to -

 
M:\tmp>7z x *.zip -o.. -y 
7-Zip 4.65  Copyright (c) 1999-2009 Igor Pavlov  2009-02-03
Processing archive: binaries_epoc_additional.zip
Extracting  epoc32
Extracting  epoc32\data
Extracting  epoc32\data\buildinfo.txt
Extracting  epoc32\data\config
Extracting  epoc32\data\config\features.dat
Extracting  epoc32\data\z
Extracting  epoc32\data\z\resource
Extracting  epoc32\data\z\resource\mtp
Extracting  epoc32\data\z\resource\mtp\102827b0.rsc
Extracting  epoc32\include
Extracting  epoc32\include\cdmalist.h
Extracting  epoc32\include\cdmaretrieve.h
Extracting  epoc32\include\cdmasmsaddr.h
Extracting  epoc32\include\cdmasmsmessage.h
Extracting  epoc32\include\gcc_mingw
...
 

• If the build from the Bill Of Materials has been selected, then you'll also see cloning of the source from the web

 
destination directory: utilities
requesting all changes
adding changesets
adding manifests
adding file changes
added 97 changesets with 192 changes to 117 files
updating working directory
42 files updated, 0 files merged, 0 files removed, 0 files unresolved
Cloning default from oss/FCL/interim/fbf/configs/default...
Cloning platforms from oss/FCL/interim/fbf/projects/platforms...
Cloning utilities from oss/MCL/utilities...
Cloning stubs from sfl/MCL/sf/adaptation/stubs...
Cloning camera from sfl/MCL/sf/app/camera...
Cloning commonemail from sfl/MCL/sf/app/commonemail...
Cloning conntools from sfl/MCL/sf/app/conntools...
Cloning contacts from sfl/MCL/sf/app/contacts...
Cloning contentcontrol from sfl/MCL/sf/app/contentcontrol...
Cloning conversations from sfl/MCL/sf/app/conversations...
Cloning devicecontrol from sfl/MCL/sf/app/devicecontrol...
Cloning dictionary from sfl/MCL/sf/app/dictionary...
...
 

• After that the build steps will begin. If you're interested in the manual steps required then either
  • Follow the link to the manual build process at the top of the Project.
  • Select the "Configure" option for a Project. This will show in detail, all of the steps required.

Success or Fail

When Helium/Hudson claim "Build Successful" - be aware that this only means that the build ran all steps to completion - but not necessarily correctly. You'll need to take a look at the vast quantity of output logs to determine if it was actually OK.

For the PDK_Helium project a good starting point would be the output\logs\html\index.html report that you'll find in your build drive. If you're using SBSv2 directly with PDK_Raptor then its not so nice - you'll just have to grep for "sbs error:" and "sbs warning:" for results.

Work is underway to make the build reporting much better, so hopefully it wont be too painful for too long.

What next?

You can explore plenty of ways to get the most out of Hudson, some include -

• Setup distributed slave machines to maximise resource utilisation.
• Install master and slave as win32 services for better robustness.

• Set HUDSON_HOME globally, if you've not already done so.

Explore - You can explore the hudson job configurations repository https://developer.symbian.org/oss/FCL/interim/fbf/hudson/.

Improve - If there are tasks or steps that you routinely do that could be turned into a Hudson job and be of benefit to the community, or you have a better way of doing things then let us know.

Contribute - Get contributing!

Open Defects

ID	P	Status	Severity	Version	Product	Summary (3 tasks) ⇒		Component
1598	P3	NEW	normal	PDK_3.0.c	Build	Hudson PDK_Raptor "RELEASE_ZIP_LOCATION" sensitive to trailing backslash	[1]	unspecified
1643	P3	NEW	normal	unspecified	Build	Helium/Hudson fail if USERNAME environment variable has space in it		build
1898	P3	NEW	Feature	unspecified	WRT Tools	Mercurial - hudson plugin integration fails.		unspecified

Further Info

• Build framework - Foundation build machinery. Explains the relationships between the different parts of the build toolchain.

Comments

Mod said…

Sign in to comment…