Build machinery
From Symbian Developer Community
Contents |
Build Machines used within Symbian
This page provides an overview of the build machinery used with the Foundation to support package and platform builds.
Physical View
This section describes the type of machines we use and how they are physically configured.
The Foundation build machines are on 2 locations. A small number of machines are physically located in the Foundation building (1BR). The majority of the machines are located off-site in a hosted environment. These 2 sites are linked by a VPN which makes them appear as one network. For a bit more discussion on this solution see my blog.
Full platform builds require a lot of processing cycles so to minimise the build time these are typically performed on powerful dedicated servers, currently 16 Core machines. For package builds and for other services processing power is not so critical. To provide flexible support for these services a pool of virtual machines are provided. These can be a mix of Linux and Windows machines, though currently the builds are primarily on windows. Currently all windows servers are running Windows Server 2003.
| Machine Types and Current Quantity | |||
| Description | OS | CPU | Quantity in use |
| Windows Physical | Win Server 2003 R2 Ent | DL585 G5 4x4 AMD | 4 |
| Windows VM | Win Server 2003 R2 Std | 4 Core (virtual) | 15 |
| Linux Ubuntu VM | Ubuntu 8.04 | 2 Core (virtual) | 2 |
| Linux Redhat | RH Ent 5 | 2 Core (virtual) | 2 |
| Esx Host | ESX | DL585 G5 4x4 AMD | 5 |
| VM Center | Win Server 2003 R2 | DL365 8 core | 2 |
Example Build Times
This section provides some example build times from builds run within the Foundation.
Single Target
We typically use the 16 core DL585 machines for full platform builds. With this type of hardware it is possible to build a single target from source in about 1 hour which provides a very useful quick test for any changes. It is possible to build the full platform on less capable machines. For PDK 2.0.d we built the full platform from source (winscw only) on a quad core virual machine (VM) and on a 16 core machine with the following times.
| Platform PDK 2.0.d - Time to build winscw emulator from source | |||
| Machine | Targets | Total | |
| GT winscw_udeb | s60 winscw_udeb | ||
| 4 Core Virtual Machine | 40 mins | 210 mins | 250 mins |
| 16 Core DL585 | 12 mins | 61 mins | 73 mins |
Full PDK creation
The table below gives an approximate breakdown of the steps and time taken to create a full PDK release. This was executed on a 16 core HP DL585. This build has two main phases, building "GT" (primarily OS layer) and building "S60" (the traditional S60 components). We are busy working towards a unified single model for Symbian^3 which would remove this legacy split. In all cases we are building for 4 targets (winscw_udeb, winscw_urel, armv5_udeb and armv5_urel).
| Platform PDK 3.0.d - Time to create full PDK | |
| Step | Duration |
| Fetch Source from Hg | 32 mins |
| Zip Source | 6 mins |
| Build Tools | 3 mins |
| Build GT (4 targets) | 48 mins |
| Build S60 (4 targets) | 202 mins |
| Analysis and post process | 69 mins |
| sbs --check | 77 mins |
| Smoke Test | 6 mins |
| Total | 7hrs 29 mins |
Note : the discrepancy between the total and the sum of the rows is due to some small minor steps not being shown for simplicity.
This is only the compilation time for winscw_udeb and assumes the source and tools are already available. In creating a PDK there are a larger number of targets (tools / armv5 /urel/udeb etc) and additional steps such as analysis and packaging. You also need to allow time to prepare the environment prior to compilation.
The Symbian^2 PDK creation for recent PDK builds has been significantly longer (up to 12 hours). This is due to some issues with using Raptor with the single "Hi Fi" model. We have some fixes and more improvements are due in next Raptor release so Symbian^2 PDK creation times should soon be restored to something similar to those shown above.
Functional View
In addition to the machines running builds there are a number of other functions required to support and automated build and test solution, as shown in the diagram above.
Build Results Database
People both inside Symbian and in the wider community may be interested in the results of the builds. There are a number of different use cases, e.g. People checking for known issues before running their own builds, Package owners helping to fix any build breaks, committers checking their changes have been built and not caused any problems etc. All build results are published to a database. This provides a summary of the build (number of errors, details of versions of source code used etc.) and also provides links to more detailed log files. The detailed log files are published through the Content Delivery Network.
Automated Test
There are currently a range of "smoke tests" executed on every platform build and additional Binary Compatibility tests executed on selected builds. These tests are currently all run on the emulator. Over time these tests will be extended to include package level tests and tests on hardware reference boards. A tool called ATS, developed by Nokia, is used to support these tests.
Content Delivery
We want all users anywhere in the world to experience fast and reliable downloads when obtaining files from the Foundation. Some of these files are also restricted to members only and may have export control considerations so the location and membership of users has to be considered. We are currently using a commercial mirroring network which helps us achieve these goals.
Build & Resource Management
There is an additional element not shown in the diagram which is the tools to automate the execution of the builds. This needs to include the ability to manage pools of machines. For example to support running package builds on all the active packages (100+) you need the ability to automatically trigger builds and automatically assign them to free machines within a pool. Currently we are using Hudson for this purpose.
Build Framework
This section gives a brief overview of the software running on the build servers and how this is used to support automation of the build and test cycle.
In progress .. sections to be added
Operating System
The build tools generally support Windows and Linux, Windows is however the primary OS supported and is the OS used when creating the PDK. It is possible to build a significant proportion of the OS on Linux but not the full PDK.
Compilers
2 compilers are used when creating a PDK:
- Winscw : Used to create the windows executable binaries used for the emulator. This compiler is delivered with the PDT.
- RVCT : Used to create the ARM binaries used for phones and also for the QEMU simulator. This compiler must be obtained from ARM.
Work is ongoing to also support GCCE see GCCE Toolchain Initiative
Make
Makefiles are central to the build system (though mostly hidden). A version of gnu make is supplied with Raptor (SBSv2) which is suitable for all single machine builds and is used for the current Foundation builds. For multi-machine clusters Electic Accelarator (EC) Make and PVMG make are both supported (see Raptor documentation for more details).
Symbian Build System
All Foundation builds use Raptor (officially known as SBSv2 "Symbian Build System version 2"). The previous build system (known generally as ABLD) is still provided and generally expected to work but is not tested by the Foundation. SBSv2 is available from the Foundation as part of the "build" package and is also included in the PDT.
Helium / Foundation Build Framework
Helium is an ANT based framework which provides a set of scripts and ANT targets to make it easier to repeat and automate builds. It is used in conjunction with the Foundation Build Framework (FBF), a set of configuration files that we've specialised to make Helium run Foundation builds.
While SBSv2 provides all the functionality required to build the Symbian platform, Helium allows us to automate the other steps that are performed as part of running a build: for example fetching the source code, preparing the environment and analysing the results.
Helium/FBF is available from the Foundation as part of the "build" package and is also included in the PDT.
ATS
ATS (Automated Test System) is, as the name suggests, a framework for automating tests. This is used within the Foundation for running automated smoke tests and dynamic Binary Compatibility tests. ATS supports execution of tests on the emulator and on hardware. The Foundation have been provided with a version of ATS by Nokia for use within the Foundation, we cannot however re-distribute this version. Nokia are in the process of making ATS open source through a site of sourceforge add link.
Hudson
Hudson is the front-end continuous integration tool that the Symbian Foundation use (and recommend) for platform and kit builds.
The tool runs each manual FBF/Helium or SBSv2 step in a pre-defined sequence, and logs console output for viewing in a web browser; with multiple slaves, the console output is aggregated back to the master webserver – allowing a centralised one-stop place for managing your builds.
Diamonds
Foundation Build Customisation and Utilities
Comments
Sign in to comment…
