From Symbian Developer Community

RAPTOR is the promotional name of the next-generation Symbian build system, also called Symbian Build System v2 (SBSv2)

Contents 1 This document outlines: 2 Fundamental technical concepts 2.1 Function Like Makefiles 2.1.1 Example of FLM interface specifications 2.1.2 Corresponding BLD.INF file fragments 2.1.3 Example of FLM body 2.2 Build configurations 2.2.1 Maintaining build configuration files 2.2.1.1 Grouping 2.2.1.2 Extending 2.2.1.3 Inclusion 2.2.2 Variants 2.2.3 Example of build configurations 2.2.4 Transient build configurations and variants 2.2.5 Combining 2.2.6 Build configuration properties summarized 2.3 Build specifications 2.3.1 Interfaces 2.3.2 Transient build configurations / variants= 2.3.3 Filters 2.3.4 Example of a build specification 2.3.5 Build specification properties summarized 3 RAPTOR architecture 3.1 Architecture 3.2 Encapsulation 3.3 Flowchart 3.4 Framework 3.4.1 Set-up file 3.4.2 Command-line 3.4.3 Plug-ins 3.4.4 APIs 3.5 Parallelization and make engines 3.5.1 Electric Accelerator features 3.6 Properties of generated makefile hierarchy 4 Platforms and setup 4.1 Windows 4.2 Linux 4.3 Electric Accelerator 4.4 Pitfalls in machine setup 4.4.1 Samba and PVM GMake 4.4.2 License servers 5 Scalability and performance 5.1 Amdahl’s and Gustafson's law 5.1.1 Amdahl’s law 5.1.2 Gustafson’s law 5.2 I/O bottlenecks 5.3 Core fraction of the build 5.4 Performance analysis 5.5 Baseline: comparison on a single Windows processor 5.5.1 Baseline: multiple processor build using SBSv1 5.5.2 RAPTOR measurements on Linux 5.5.2.1 Test series: constant code size with growing machine size 5.5.2.2 Hardware 5.5.2.3 Software 5.5.2.4 Test series: growing code size with constant machine size 5.5.3 Conclusions 5.6 The performance impacts of extensions to RAPTOR 5.6.1 Template Extension Makefiles 5.6.2 Function Like Makefiles 6 Benefits 6.1 Short-term benefits 6.2 Incremental builds (speed) 6.2.1 Parallel builds (speed) 6.2.2 More flexibility 6.3 Mid-term benefits 6.3.1 Integration into IDEs 6.3.2 Product-line engineering in production environments 6.3.3 Integrating other phone software production use-cases 7 Migration to RAPTOR 7.1 Steps taken to ease migration 7.1.1 Backwards compatibility 7.1.2 Binary equivalence 7.2 What needs to be done to migrate to RAPTOR 7.2.1 Migrations for all customers 7.2.1.1 Build scripts 7.2.1.2 BR.1968 RAPTOR produces a different logging output 7.2.1.3 Missing dependencies 7.2.2 Migrations that apply to customers that use legacy SBSv1 functionality 7.2.2.1 Extension makefiles 7.2.2.2 ABIv2 7.2.2.3 Old build platforms 7.2.2.4 BR.1967 RAPTOR does not use BSF files 7.2.3 Migrations that apply to Linux only 7.2.3.1 Preparing the code base for Linux 7.2.3.2 Preparing Template Extension Makefiles for Linux builds 7.2.4 Migrations for optimization purposes 7.2.5 Replace TEMs with FLMs= 7.2.6 Migrations that affect customers who use internal SBSv1 interfaces 7.2.6.1 BR.1964 RAPTOR produces different makefiles 7.2.6.2 BR.1966 RAPTOR will not include the makmake tool

This document outlines:

• the fundamental technical concepts on which SBSv2 is built
• the architecture of SBSv2
• the platforms on which SBSv2 runs (Linux and Windows)
• scalability of SBSv2
• the short-term and long-term benefits
• what needs to be done by customers to make use of SBSv2 in their build environment.

Fundamental technical concepts

This section describes the fundamental technical concepts that underpin RAPTOR.

Function Like Makefiles

The concept of Function Like Makefiles (FLMs) is at the core of RAPTOR and has been invented to solve some of the restrictions that exist in GNU Make today. Restrictions in GNU Make exist in the following areas:

• modularity and code re-use
• separation of interface and implementation
• extensibility
• testability.

FLMs solve these problems through a combination of meta-information describing an FLM and coding conventions. FLMs do this without requiring changes to the GNU Make language and thus changes to the make engines supporting these. Their key function is to reduce complexity and to make it possible to create a makefile-based build system that is maintainable, while also achieving good scalability in builds.

FLMs contain two parts:

• an interface specification part (encoded in XML)
• an implementation part (or body) which is encoded in GNU Make.

The specification part of the FLM exposes inputs, outputs, configurable parameters, and meta-information of the FLM. This makes an FLM an independent, re-usable, and testable component encoded in GNU Make. In essence the FLM becomes a black box with inputs, outputs, parameters and specified behaviours, which RAPTOR combines into a large makefile.

1. Note that for historical reasons the name RAPTOR will be used instead of SBSv2 throughout this document.
2. In fact similar restrictions exist for most make / build engines.
3. Note that currently RAPTOR treats inputs, outputs, and meta-information like parameters.

Another way of trying to understand FLMs is to think about them in terms of C++ templates, with the key difference that they do not instantiate a class or function, but rather, a section of a dependency tree. As in a higher-level language, the templates are the building blocks to construct a larger system.

Coding conventions are used to achieve the make equivalent of re-entrancy (in other words, the same FLM can be included in the same hierarchy of makefiles several times). FLMs can make use of other FLMs, and therefore, complex FLMs can be created from simpler ones. All build processes in RAPTOR are implemented as FLMs, from core build processes used to build Symbian OS binaries, through to special functionality build processes such as kernel bootstraps. Therefore, extensibility is at the heart of RAPTOR.

In opposition to most languages where interfaces and implementation are separated, currently RAPTOR does not allow several FLM bodies to implement the same interface4. The reason for this is that this functionality was not needed in RAPTOR, and that the separation of interface and implementation was introduced to:

• make it easy for the RAPTOR back-end to assemble larger makefiles from FLMs
• ensure that RAPTOR needs to know as little as possible about makefiles
• enable tooling that can be used to verify the FLM body against the FLM interface specification.

This means that it is possible to replace one FLM describing a build process with a different one, with little or no impact on the overall build system. Bugs in the build system can be detected easily, as each FLM can be tested individually. This means it is possible to isolate faults. Bugs can also be detected through tooling that supports formal verification of FLMs.

Example of FLM interface specifications

The following section shows an example of FLM interface definitions and how these use derivation. Currently, all inputs and outputs in an FLM are treated equally.

 
<?xml version="1.0" encoding="ISO-8859-1"?>
 
<build xmlns="http://symbian.com/xml/build"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://symbian.com/xml/build ../../schema/build/2_0.xsd">
 
       <interface name="Symbian.flm" abstract="true">
              <param name='FLMHOME' />
       </interface>
       <interface name="buildprogram" extends="Symbian.flm" flm="buildprogram.flm">
              <param name='TARGET' />
              <param name='SOURCEFILE' />
              <param name='GCC' />
              <param name='DEBUG' />
       </interface>
       <interface name="strippedprogram" extends="buildprogram" flm="strippedprogram.flm">
              <param name='STRIP' />
       </interface>
</build>
 

The interface Symbian.flm is an abstract type, which is the root of all FLM interfaces that are used in RAPTOR. RAPTOR uses a hierarchy of abstract FLM interfaces for all FLMs to ensure consistency in core RAPTOR functionality. For example Symbian.MMP is the interface for all objects that are described in MMP files, Symbian.DLL is the FLM interface for all DLLs, Symbian.EXE is the FLM interface for all EXEs, etc. Inevitably there will be different implementations of FLMs for each interface, for example, a different DLL implementation for each toolchain.

Corresponding BLD.INF file fragments

Declaring FLMs with interface XML files like the above allows the following syntax to be added to BLD.INF files, inside a PRJ_EXTENSIONS section:

PRJ_EXTENSIONS

START EXTENSION buildprogram
TARGET my.o
SOURCEFILE my.cpp
END EXTENSION

START EXTENSION strippedprogram
TARGET my2.o
SOURCEFILE my2.cpp
END EXTENSION

The parameters TARGET and SOURCEFILE are declared with <param> tags in the interface XML, and these are supplied in the BLD.INF file. The remaining parameters, GCC, DEBUG, and STRIP must be defined in the configuration.

The extension block is called buildprogram because that is the name declared in the <interface> tag in the interface XML. The .flm file itself can have a completely unrelated name if you wish.

Example of FLM body

The following section contains some simple examples of FLM bodies that match the interface specification above. The include files define global constants and some GNU Make macros for common operations. For example, constants for host specific characters, macros to create output in a consistent manner, macros to create standard boilerplate code for creation of paths or cleanup, etc.

# buildprogram.flm
#
# Copyright (c) 2007 Symbian Ltd. All rights reserved.
#
# Function Like Makefile (FLM)
# Builds a simple program using the GNU Toolchain

define buildprogram1

$(TARGET): $(SOURCEFILE) | $(CREATABLEPATHS)
         $$(call startrule,buildprogram2) \
         $(GCC) -o $$@ $(if $(DEBUG),-g,) $(SOURCEFILE) \
         $$(call endrule,buildprogram2)
endef

# Evaluate - expand all variables so that a unique rule is created.
$(eval $(call buildprogram1))

## Create paths -- you may add to the paths to be created here
$(eval $(call GenerateCreatablePathTargets,$(CREATABLEPATHS)))
## Clean up
$(eval $(call GenerateStandardCleanTarget,$(TARGET) $(CREATABLEPATHS)))
## What targets -- declare the files produced here
$(eval $(call whatmacro,$(TARGET)))

In the examples above and below, all the text in italics is common boilerplate code that deals with output, creating directories, and cleaning up make targets. The code in bold is the actual build target, which uses macros in order to make the FLM look more consistent.

Notes:

• note the names of the parameters from the XML file above appearing in the FLM
  • TARGET and SOURCEFILE come from the BLD.INF's START EXTENSION buildprogram block
  • GCC and DEBUG come from the configuration
    • There is no essential reason why GCC and DEBUG could not be set in the BLD.INF file instead, but this is clearly not the intention
  • You are permitted to use these four variables because you declared them in <param> tags your interface XML
  • If any of these variables are defined neither in the BLD.INF file nor the configuration, the build will fail with an appropriate error
• buildprogram1 is a completely arbitrary name
• buildprogram2 is what this rule will be called in the build log
• CREATABLEPATHS contains a list of directories that will need to be created; you do this with the 'Create paths' line
• the whatmacro function call sometimes takes a second parameter. This was mainly for historical reasons, and can now usually be left out.

The following example shows how an FLM calls another FLM. In this example, the strippedprogram FLM removes all symbolic information from a binary that is built by the buildprogram FLM.

# strippedprogram.flm
#
# Copyright (c) 2007 Symbian Ltd. All rights reserved.
#
# Function Like Makefile (FLM)
# Builds a simple program using the GNU Toolchain and strips any symbolic information
# (including debug info)

STRIPPEDTARGET=$(TARGET).strip

define strippedprogram

$(STRIPPEDTARGET): $(TARGET) | $(CREATABLEPATHS)
        $$(call startrule,strip) \
        $(STRIP) -o $$@ $(TARGET) \
        $$(call endrule,strip)
endef

# Evaluate - expand all variables so that a unique rule is created.
$(eval $(call strippedprogram))

# Call the buildprogram FLM to ensure that the program is built
include buildprogram.flm

## Create paths
$(eval $(call GenerateCreatablePathTargets,$(CREATABLEPATHS)))
## Clean up
$(eval $(call GenerateStandardCleanTarget,$(STRIPPEDTARGET) $(CREATABLEPATHS)))
## What targets -- declare the files produced here
$(eval $(call whatmacro,$(STRIPPEDTARGET)))

Because FLMs separate interface and implementation, it is possible to develop test code against the interface specification. This is done by creating test vectors for inputs and parameters using the interface specification. The FLM is called with the test vectors as input. The test checks whether the correct outputs have been produced, and whether the FLM invokes command line tools in the right sequence with the right parameters.

In terms of granularity, RAPTOR uses FLMs for parts of the build which are independent of each other, and as large as possible. For example, RAPTOR uses FLMs for building bitmaps from sources, and for building Symbian platform binaries such as DLLs and EXEs from sources.

Build configurations

Another key generalization in RAPTOR over SBSv1 is the build configuration. Build configurations are in essence vectors of (parameter, value) pairs that control the build. These (parameter, value) pairs provide the values for the parameters that are exposed by all the FLMs that are needed to perform a build.

Build configurations which provide global default values for non-transient FLM parameters are stored in XML files and called build configuration files. These files normally describe build configurations that are linked to a specific toolchain or target type such as the GCC, ARM or the WINSCW toolchains.

Most parameters that influence the build directly originate in FLMs, in other words, they are parameters that are explicitly used in the FLM body, such as $(DEBUG) in the example above. Some are indirect, such as environment variables that are required by a tool that is called from an FLM, such as $RVCT22LIB, which is needed by the Real View Compiler. For this reason, build configuration files allow specification of dependencies on environment variables, optionally with default values. This enables RAPTOR to verify whether the build environment is complete, and return a concrete error should this not be the case, or provide sensible default values where appropriate.

Maintaining build configuration files

Managing lists of (parameter, value) pairs is made easier in RAPTOR by using the following concepts.

Grouping

More complex build configurations can be constructed by grouping simple build configurations into a hierarchy. To enable this, it must be possible to name build configurations and use them in different combinations.

Extending

For compactness of representation and ease of maintenance, it is possible for a build configuration to “extend” one named build configuration. This is similar to derivation as used in object-oriented languages such as C++.

Inclusion

Besides extension, inclusion can be used to group named build configurations into a new build configuration. Inclusion is used in RAPTOR for build configurations that are static, and often at the root of a hierarchy of build configurations. In some sense, inclusion is used to implement a simple form of multiple derivation (not supported by extensions) that avoids the problems of multiple inheritance.

Variants

The concept of variants is central to RAPTOR. For the end-user, a variant is a different version of a build. For example, PAGED could be a variant of the standard ARMv5 build with Demand Paging functions turned on.

In general a variant is something which modifies a build. As discussed earlier, build configurations control the behaviour of a build by providing the parameters to the tools that are executed as part of the build process. Therefore, a variant of a build can be seen as a set of functions which change these parameters, or in other words, a set of functions that change build configurations.

Derivation of build configurations already allows overriding of values in the parent build configuration. By allowing the following operations in build configurations, build configurations can be used to describe changes to other build configurations:

• appending a value to the end of a variable
• inserting a value at the beginning of a variable
• deleting a variable
• list operations.

This means that a variant becomes a special type of a build configuration. Build configurations that define default values for build processes are not considered variants, while build configurations that change the default are considered variants. In practice, there is no real difference between a build configuration and a variant.

Example of build configurations

The following example shows small pieces of the build configuration which implements the ARMv5 Build Target in SBSv1. The example shows also how derivation is used to create the UREL and UDEB variants of the ARMv5 build.

Transient build configurations and variants

Build configuration files only describe non-transient build parameters in FLMs. Transient parameters, such as input and output files, are specific to a component, and therefore cannot be separated from components. Transient build parameters are generated as needed, based on information in input files such as MMP and bld.inf files. They are stored within RAPTOR in the same data format as build configurations, though only in RAPTOR’s internal data model.

Some MMP file keywords such as OPTION and OPTION_REPLACE allow changing the default behaviour of builds. Changing the default behaviour of builds is equivalent to changing default variables as expressed in build configuration files. The changes to the default are translated into transient variants, in other words, build configurations that exist only in RAPTOR’s internal data model and modify default parameters.

The key difference between transient and non-transient build configurations, is that non-transient configurations are typically applied to all components in the system and stored in XML files, whereas transient build configurations are only applied to a part of the system (usually only individual components), and are stored in RAPTOR’s internal data model. When we talk about build configurations, we normally refer to non-transient configurations.

Combining

RAPTOR allows the combining of two different trees of build configurations, in a similar way to the concept of “cross products” in linear algebra. The intention of this is to make it easy to apply variants to other variants or build configurations.

Two hierarchies of build configurations

In this example, let us assume the configuration ABC describes “ARMv5 UDEB” and “ARMv5 UREL”, and the configuration DEF describes the different feature sets “Demand Paging ON” and “Demand Paging OFF”. Combining the two variants gives the following graph:

Combination of two hierarchies of build configurations

This approach enables the easy application of the variants “Demand Paging ON” and “Demand Paging OFF” on top of “ARMv5 UDEB” and “ARMv5 UREL”.

Build configuration properties summarized

Build configurations are a generalized way of describing parameters for FLMs. Build configurations have been designed such that they are:

• extensible by anybody in the ecosystem, without the need to change Symbian OS source code
• easy to use
• easy to maintain.

The reason for this is so that key users in the ecosystem are likely to want to adapt build configurations to their needs. It is also planned that build configurations replace some of the file formats in SBSv1, for example, BSF files.

Build specifications

RAPTOR uses build specifications in its internal data model to pull together all non-default information that is needed for a build. It does this in a form that easily allows the creation of large makefiles. Build specifications expose a hierarchy of build steps at different levels of granularity: at the top level they expose the hierarchical structure of the system (as defined in System_definition.xml), the hierarchical structure within bld.inf files (i.e. MMP files contained within bld.inf files), and the implicit hierarchical structures that are contained in MMP files. In other words, a build specification describes the hierarchy of the entire system, from the system definition, down to individual FLMs. In addition to this, each node in the hierarchy contains information that is relevant to it, such as transient build parameters. Or put more simply: build specifications describe what to build and how build parameters deviate from the default.

Currently, build specifications only exist in RAPTOR’s internal data model. However, in future Symbian OS releases, build specifications may become the main interface into the build tools for some use-cases. For example, an IDE may interface with the build specification rather than an MMP file.

Interfaces

An important property of a build specification node is a reference to the FLM interface which defines the build process to be used to build the node. Some nodes are used purely for grouping and have no interface. All other nodes contain exactly one active interface.

Transient build configurations / variants=

The build specification also describes how build parameters deviate from the default for a given node. This includes the following two cases:

1. Transient parameters for which no default values are sensible, such as source files.
2. Parameters which are overridden for a particular target or component. An example is the OPTION keyword in MMP files, which can be used to change arguments that are passed to the compiler or linker.

 
OPTION CW –w off
 

This example switches off warnings for all invocations of the CodeWarrior compiler for the component described by an MMP file. In both cases, the parameters are represented using transient build configurations (which are also variants). This means that all parameters that are passed to FLMs for the build are simply calculated by finding the build configuration that describes the default values and applying all relevant variants to it in the right order. It does not matter whether the variant is a named build configuration or embedded in a build specification.

Filters

The build specification tree can contain conditional elements, in other words, the build specification tree contains nodes and variants which only become active when used in conjunction with certain build configurations or variants.

This conditional structure is represented using filters. A filter is a “wrapper” around a set of nodes, or a set of variants, or an interface which makes statements such as

• this part of the tree is only valid for configuration X [and optionally derived configurations]
• this part of the tree is only valid for configurations containing variant Y
• this part of the tree is not valid for configuration X or configurations containing variant Y.

Filters will have an “else” part (or an inverse), so you can specify “use this if the filter is satisfied” otherwise “use this if the filter is not satisfied”.

Multiple filters can be applied. Conditional parts of a build specification are valid (and thus in existence) if all the filters “surrounding it” are satisfied. This means that one build specification can describe a family of software products that is created from the same code base. As mentioned earlier, a build specification node can have at most one active interface. This means that several interfaces may be defined for a node, but when all conditions are resolved, only one active interface is instantiated. Violation of this rule results in errors.

Example of a build specification

The following example shows sections of a build specification that has been generated while building an example component in Symbian OS. Note that the example does not show filters.

The example shows the hierarchy of build specification nodes that have been generated from a component described by a bld.inf file, and how variants (which are a form of build configuration) are used to describe parameters that are specific to the component. The other set of parameters are encoded in a build configuration, which are more global, and in this case are applied to all build specification nodes.

Build specification properties summarized

Currently, build specifications are only machine-generated, and not intended for manipulation by end-users of RAPTOR. For this reason, not much consideration has been given to usability of the build specification file format.

RAPTOR architecture

RAPTOR is designed to create, maintain, and extend a high-level model of multiple builds. It uses this model to generate a hierarchy of makefiles. The generated makefiles are based on the concept of Function Like Makefiles, which extends on the concept of Template Extension Makefiles (TEMs) in SBSv1.

RAPTOR automatically constructs a complex makefile (containing a complete dependency tree) by combining prefabricated FLMs using simple “glue code” to connect them.

Architecture

The diagram below shows a schematic of a generic build system. A build is basically a file transformation process. A number of source files are translated, often iteratively, into a number of target files, which are the build outputs. The build system uses toolchains to create different products for different target platforms.

Generic build model

In the diagram, the generic model does not use the exact definitions as given earlier. Instead, a build specification is used to describe “what to build”, while the build system configuration is used to describe “how to build”.

Encapsulation

One of the key problems with SBSv1 is that the original design did not clearly encapsulate functionality. This makes SBSv1 hard to use and maintain, inefficient to run, and hard to extend.

RAPTOR uses encapsulation rigorously and consists of only a few logical parts. This means very few parts of RAPTOR are complex. The following table decribes the complexity in RAPTOR in more detail.

Element	Description
RAPTOR front-end	The purpose of the RAPTOR front-end is to parse the meta-data input files (System_Definition.xml, bld.inf and MMP files) in the context of one or more Symbian OS versions and translate their structures into generic RAPTOR data structures. Today a significant proportion of the complexity of RAPTOR is in the front-end. The front-end handles all the special casing and the pre-processor directives in bld.inf and MMP files. All the complexity that has been built over time in SBSv1 is carried over into the RAPTOR front-end, and related processing is explicit in one place.
RAPTOR data model	The RAPTOR data model represents the following generic concepts and their relationships: Build configuration – defines the default arguments that are passed to individual elements of the FLM library. Build specification – defines what to build, and deviations from the default. FLM library – a library whose elements encapsulate how to translate input files into built artefacts up to the last level of detail. The data model has been designed so that it is generic and independent of the front-end and can be represented in text form (XML). If Symbian were to deprecate bld.inf and MMP files at some point in future, this would remove the need for complex processing in the front-end.
Function Like Makefiles (FLMs)	This is an individual element of the FLM library, which encapsulates how a set of input files is translated into a set of output files. FLMs are implemented in GNU Make notation in RAPTOR. However they could be implemented in any “language” that is used to build applications. FLMs contain most of the complexity within the build system.
RAPTOR back-end	The RAPTOR back-end generates a hierarchy of makefiles from the data model. It does this by combining FLMs to a larger makefile and organizing information that is provided in build specifications and build configurations in the correct FLM. Make engine The make engine is the vehicle that is used to perform the actual build. RAPTOR supports all make engines that fully support the GNU Make file format, such as GNU Make, PVMgMake, and eMake (from Electric Cloud).

Flowchart

The flowchart below illustrates the basic data-flows in RAPTOR. The key components (the front-end, the data model and the back-end) have been described above. RAPTOR constructs build specification from files that SBSv1 or other Symbian tools have introduced in the past. These files describe

• the system via the System_definition.xml file
• components via bld.inf files
• executables via MMP files.

RAPTOR does this to be as compatible with SBSv1 as possible. However, in future releases, RAPTOR may be able to consume new file formats, should this be necessary. The RAPTOR architecture makes it possible to replace bld.inf and MMP files, enabling a gradual migration from SBSv1 file formats to new formats.

Framework

RAPTOR itself is implemented as an extensible framework, as illustrated below. The framework and all but the make engine are implemented in Python8. There are three main ways to drive RAPTOR:

1. Directly from the RAPTOR command-line.
2. Indirectly from the ABLD command-line which provides compatibility with SBSv1.
3. Directly from an IDE using the RAPTOR API. An IDE could also drive RAPTOR via the command-line, rather than using the API.

RAPTOR framework

The RAPTOR Framework exposes a plug-in architecture that allows the loading of different plug-ins to support alternative command line interfaces, input file formats, and make engines. The framework is configured using an XML file – the init.xml or set-up file. File parser plug-ins are used to process Symbian OS specific file formats such as MMP files. Core RAPTOR functionality is exposed through the RAPTOR API which can be used to drive RAPTOR from an IDE.

Set-up file

The root directory of RAPTOR ($RAPTOR_HOME) contains the RAPTOR initialization file (init.xml) which controls the default behaviour of RAPTOR. The file specifies locations of plug-ins, locations for configuration files such as build specifications and the FLM library, and other defaults, for example, the default build configuration and the default system definition. The following example shows a RAPTOR initialization file.

Command-line

The primary way to drive RAPTOR is the command line. There are two scripts in the root directory of RAPTOR for launching the RAPTOR executable and passing it the correct arguments:

• sbs for Linux
• sbs.bat for Windows.

Plug-ins

RAPTOR supports several types of plug-ins:

• parsers for SBSv1 files
• command-line interpreters
• make engine drivers.
• log file parsers

These plug-ins may require customization by licensees or replacement at a later date. For this reason, they have been isolated by clearly defining an interface between the plug-ins and the main RAPTOR code.

APIs

RAPTOR is implemented in Python, which is an object-oriented scripting language. The framework, plug-ins and data model can all be accessed directly using the public APIs of the objects which represent them. Common functionality is hidden behind the RAPTOR API.

Parallelization and make engines

One of the key features of RAPTOR is its support of parallel building on appropriate hardware. As RAPTOR relies on a third party make engine for parallelization, its parallelization capabilities are “inherited” from the make engine. The GNU Make 3.81 standard was chosen because it provides the most options with regards to parallelization on different hardware platforms. The make engines that comply with the GNU Make 3.81 specification are listed below.

Make engine	Main user	Linux	Windows	Tested by Symbian
GNU Make	Developer	Yes	Yes	Windows and Linux GNU Make is tested in single CPU and multi-CPU mode (using the –j option)
PVM GMake (open source)	Build Team	Yes	No	Linux
Electric Accelerator	Build Team	Yes	yes	Windows and Linux

Electric Accelerator features

Electric Accelerator is the most comprehensive make engine in the table above, and provides the range of features listed below.

• A caching file system.
• The capability to fix dependency trees that are incomplete or otherwise broken. This feature will help in situations where a large amount of the build is performed from within extension’s makefiles, for example, third party Java code.
• The capability to create a large dependency tree from smaller disjointed individual trees (representing individual makefiles). Again this feature will help with extensions.
• Load balancing capabilities, which lead to increased performance by taking the time it took to build individual artefacts into account in subsequent build runs.
• Additional analysis and build management tools.
• Fail safety and redundancy, in other words, the build will complete even if hardware or software on the build cluster fails.
• The capability to manage the software that is installed on nodes within the cluster using virtualization.

Properties of generated makefile hierarchy

To enable parallelization on all the make engines listed above, makefiles generated by RAPTOR must have the following key properties.

• The makefiles must encode a complete and correct dependency tree. Without a complete and correct tree, parallel builds can fail randomly. Generally, RAPTOR achieves this without changes to source files, and deduces dependencies from various information sources in the build. However, in some rare cases it is necessary to augment MMP files with additional dependency information by using the DEPEND keyword (see section 6 for more detail). Also note that extensions to RAPTOR using the Template Extension Makefile mechanism may lead to incomplete or incorrect dependency trees if the extension is encoded incorrectly. As TEMs were originally developed by most of Symbian’s customers using SBSv1, where correct dependency trees were not required, dependency issues with TEMs written for SBSv1 are common. Due to RAPTOR’s design, missing dependencies in TEMs will not affect parallel builds, but can affect the correctness of incremental builds.
• One large makefile must be generated, except in some situations, such as for SBSv1 template extension makefiles that are supported by RAPTOR.
• All build artefacts must be unique and must use absolute paths.

Besides enabling parallel builds, these makefile properties are also sufficient to enable incremental builds.

Platforms and setup

The following section gives a brief overview of the test set-ups that Symbian is using.

Windows

Symbian uses a HP XW 4400 with the following specifications for individual team builds:

• dual-core Pentium CPU, 3.4Ghz
• 2GB RAM
• 160 GB HDD.

Symbian uses a HP ProLiant DL 585 with the following specifications for system builds:

• four dual-core AMD Opteron 875 CPUs, 2.2 GHz
• 8GB RAM.

RAPTOR is tested for functional completeness on both machines using GNU Make without –j. However, only a few test runs with –j will be performed to assess performance.

The reason for not focussing performance testing on Windows is that file system performance on Windows is significantly worse than on Linux (based on earlier studies), but the Electric Accelerator caching file system addresses this.

Linux

Symbian uses an HP blade cluster with the following specifications:

• As head node, a HP ProLiant DL380 G5 Server with 3GHz dual-core Xeon 5160 processors, with 2GB RAM, and 700GB storage.
• An 8 dual-core HP ProLiant BL465c 2220 DC (1P) with 2.8 GHz AMD Opteron 2220 processors, with 4GB RAM, and 80GB storage.
• An 8 quad-core HP Quad Core Intel 5355 Blade Server (1P) with 2.66GHz Intel Xeon 5355 processors, with 4GB RAM, and 80GB storage.

RAPTOR is tested for performance primarily with PVM and PVM GMake, and secondarily with GNU Make –j. The head node is not used for builds.

Symbian chose a mixed machine set-up, to be able to identify the optimal machine configuration, and determine whether dual-core processors are better than quad-core processors for RAPTOR, even though this introduces complexities for some performance measurements.

Electric Accelerator

Compliance and performance testing with Electric Accelerator will be performed in partnership with Electric Cloud. Data will only be made available on demand on a case-by-case basis. The reason for this is that Symbian platform licensees’ build environments are significantly more complex than Symbian’s internal build environment. For example, the number and size of build operations performed in extension makefiles is minimal within Symbian’s code base, but can be large for licensees, for example, Java code.

Pitfalls in machine setup

Symbian experienced various problems with parallel test runs at various points during the development cycle.

Samba and PVM GMake

During early development, Samba was used to work around filename case issues on Linux. This approach had worked for test runs without PVM. Test runs using PVM GMake failed due to Samba “disappearing” when PVM GMake was run.

License servers

When Symbian started testing RAPTOR, it had significant issues with FlexLM and RVCT, which limited performance testing on the Linux Blade clusters. Symbian has a policy of using floating licenses for most tools and it is difficult to get node locked licenses. The recommendation for RAPTOR is to use node locked RVCT licenses, or to run a license server on the cluster.

The issues appear in random build failures due to too many license requests to the license server in a short period of time, and as a result, RVCT cannot obtaining a license. Using re-try options helps but does not solve the problem, and the problem gets worse with the number of processors that are used during a build, which clearly indicates a bandwidth issue.

Scalability and performance

Amdahl’s and Gustafson's law

Two mathematical laws govern scalability in parallel computing

1. Amdahl’s law, which deals with the relationship between the number of processors and the speedup that can achieved on a parallel system, while keeping the problem size constant.
2. Gustafson’s law which looks at the speedup one can achieve on a parallel system, assuming that the problem size grows with the size of the parallel system.

Both laws are relevant for predicting and measuring the scalability of RAPTOR on parallel build clusters and for this reason, are stated in some detail in this document.

Amdahl’s law

Amdahl’s law is used in parallel computing to predict the theoretical maximum speedup of a problem. The speedup S is defined as wall-clock time of best serial execution divided by the wall-clock time of parallel execution.

For parallel problems, Amdahl’s law states that if F is the percentage of a calculation that is sequential (also called the serial fraction, or the fraction of a computation that cannot benefit from parallelization), and (1 − F) is the fraction that can be parallelized (also called the parallel fraction), then the maximum speedup that can be achieved by using N processors is:

Where we have set a total time of F + (1 − F) = 1 for algebraic simplicity. For large systems, this is an extremely steep function near F = 0. At the limit, as N moves towards infinity, the maximum speedup moves towards 1/F. In fact the graph implies that very few problems will experience even a 100-fold speedup, as only very few problems will have a sufficiently small serial fraction.

The law contains the implicit assumption that the serial fraction F of a problem is independent of the number of processors N, which is virtually never the case. One does not take a fixed-size problem and run it on various numbers of processors except when doing academic research. In practice, the problem size increases with the number of processors. Amdahl’s law is also problematic as it is difficult to estimate the serial fraction F of a problem.

Gustafson’s law

Gustafson’s law governs the behavior of “sufficiently” large problems which are parallelized. It addresses the short-falls of Amdahl’s law. The key difference is that the serial fraction of a problem is not considered independent of the size N of the parallel computer on which the problem is solved. It is also a function of the problem size n.

The serial fraction F(n) is also called serial function. Assuming that the serial function diminishes with growing problem size, the speedups approaches the maximum speedup that can be achieved on a particular parallel computer. In practice this means that if we can show that the serial function of a problem decreases with the problem size n, we can prove that throwing more processing power at the problem will lead to increasing speedups.

I/O bottlenecks

I/O throughput is likely to be a key factor in limiting the scalability when parallelizing builds. Bottlenecks may be caused by the following factors:

• the performance of the network for I/O operations that use NFS
• the speed of reading from, and writing to disks.

Both of these factors can contribute to the serial fraction of the problem size and are parameters that can be changed (up to a degree) by choosing build hardware with faster disks, or a faster network.

Core fraction of the build

The time it takes RAPTOR to create makefiles from input files is called the core fraction of the build. The core fraction is executed serially on one CPU. The actual building (in other words, execution of make) can be performed in parallel. The overall build time is the sum of the two times.

Currently Symbian has no plans to parallelize the core fraction of the build, but may choose to do so in future.

Performance analysis

This section contains RAPTOR performance measurements.

Baseline: comparison on a single Windows processor

This section compares RAPTOR against SBSv1 on a single processor (not using hyper-threading). This comparison is only available on Windows and has been performed on a standard HP XW 4400 desktop machine with a 3.4 GHz processor. The table shows measurements for a single component, which in this case is multimedia_mmf.

	SBSv1	RAPTOR
Run 1	4.6 minutes	4.0 minutes
Run 2	4.9 minutes	4.0 minutes
Run 3	4.8 minutes	3.9 minutes
Average	4.8 minutes	4.0 minutes

Without exploiting parallelization, RAPTOR is approximately 20% faster than SBSv1. Other components show similar performance improvements.

We have not performed a wide range of performance tests on single processor machines, which may mean that the measurements are not valid for a growing problem size.

Baseline: multiple processor build using SBSv1

The complete Symbian OS v9.5 build time on a 4 dual-core 2.8Ghz AMD 64 processor, with 8GB RAM, using Windows, is 116 minutes. This time includes the following build configurations: ARMv5 UREL, ARMv5 UDEB and WINSCW UDEB. Note that these are slightly faster specifications than the ones defined in section 4.1, to allow direct comparison with builds performed on the Linux cluster. Note that due to its architecture SBSv1 does not scale beyond 8 CPUs.

RAPTOR measurements on Linux

Test series: constant code size with growing machine size

The measurements covered in this section are for a complete Symbian OS v9.5 build (ARMv5 UREL, ARMv5 UDEB and WINSCW UDEB) using RAPTOR while increasing the number of CPUs on the Linux build cluster.

The tests shown below do not show the usage of hyper-threading on individual CPU cores during the build, as previous tests have not shown any benefits. Hyper-threading means that several artifacts are built on one core at the same time.

The tests have been performed by configuring the cluster with different hard-disk setups: standard (no RAID) and data striping (or RAID 0). I/O performance for the latter is higher than in the standard setup, which allows us to see the impact of different disk performance on the build.

The following graph shows the SBSv1 baseline on eight CPUs, and two series of tests using different disk arrays. The tests exclude the core fraction of the build, which in this case is 8 minutes.

ARMv5 (UREL and UDEB) and WINSCW build time for Symbian OSv9.5 on a Linux blade cluster

As the problem size is fixed, we are seeing the effects of Amdahl’s law, and the diminishing returns for a larger number of CPUs, as expected. Analysis of the build times shows that in our hardware set-up, I/O performance (in this case the disk speed) is the bottleneck:

• A faster disk increases the performance of the build by 18% (No RAID vs. RAID 0).
• For I/O intensive build steps such as linkage, individual CPUs generally wait for disk operations.
• Build configurations which require fewer disk accesses are faster than build configurations which require more disk accesses. It takes five minutes to perform an ARMv5 UREL build, 10 minutes to perform a WINSCW UDEB build, and 15 minutes to perform an ARMv5 UDEB build on 46 CPUs (no RAID). This agrees with an increased I/O wait proportion for more I/O intensive build configurations. The difference between the various build configurations is the amount of debug information that is generated by the compiler, and consequently has to be read by other tools such as the linker.

In the first release of RAPTOR, no optimization opportunities beyond the use of parallel make engines have been used. All disk operations in RAPTOR are performed via NFS. However, analysis has shown that there is plenty of room for future optimizations in hardware and software:

Hardware

• higher performing disks and disk controllers
• faster NICs
• RAM disks.

Software

• more node-local operations to reduce I/O traffic
• specific optimizations for build acceleration tools such as Electric Accelerator. Symbian will work with Electric Cloud to explore and implement such optimisations in future.

Test series: growing code size with constant machine size

We ran a series of tests in which we increased the problem size while keeping the machine processing size constant. Using the Symbian OS source code as the basic unit of “code size”, five builds were performed, each one containing a multiple of the amount of code in Symbian OS (in other words, build 1 = 1 x Symbian OS, build 2 = 2 x Symbian OS, … , build 5 = 5 x Symbian OS). The size of the cluster and the number of allowed parallel tasks were constant throughout (28 CPUs and 28 tasks).

RAPTOR was used to create one makefile for a single build of the Symbian OS source code. Five copies of this makefile were created and the copies were adjusted so that the targets for each makefile would be created in different locations. A wrapper makefile was created that included these manually modified makefiles13. This approach ensured that all targets in the makefile were treated as if they were different. The following graph shows the increase in build time, which as expected, is nearly linear.

Build time vs. growing source base

This indicates that for five times the size of the Symbian OS source code, we are not exposed to any additional hardware bottlenecks.

Conclusions

We have shown that it is possible to build Symbian platform in approximately 30 minutes (including the core fraction of the build), and customer builds are likely to experience similar performance increases, depending on the size of their code base. There is also significant potential for further optimization depending on the hardware and software used.

The performance impacts of extensions to RAPTOR

Extensions to RAPTOR enable Symbian’s customer to add build steps that are not supported by the standard version RAPTOR. RAPTOR supports two different extension mechanisms:

• Template Extension Makefiles (also supported by SBSv1)
• Function Like Makefiles.

Symbian cannot guarantee that build extensions to RAPTOR encode complete and correct dependency information.

Template Extension Makefiles

TEMs are supported in RAPTOR to allow Symbian’s customers to smoothly migrate their code bases to RAPTOR. Customer code bases will have a significant number of TEMs in their code base. RAPTOR cannot parallelize extensions that use TEMs: they are executed using nested calls to the make engine on individual CPUs to guarantee the correctness of the build. If a large proportion of the build is encoded using TEMs, RAPTOR cannot realize its parallelization potential in extensions. If large components (for example, an implementation of Java) are integrated into a RAPTOR build using TEMs, the overall build time is limited by the time it takes to build these large components.

Make engines such as Electric Accelerator are designed to resolve such issues by correcting missing or incorrect dependencies in TEMs, and by overcoming the boundaries that are introduced by nested calls to make engines. The performance of RAPTOR will increase significantly in such situations.

Function Like Makefiles

FLMs are designed to overcome the limitations of TEMs. However, they require the migration of build extensions to a new framework. For RAPTOR to work correctly in parallel and incremental build situations, FLMs also need to capture all dependencies correctly. Depending on the amount of extensions needed in a customer environment, migrating extensions to FLMs may require a significant amount of effort. In addition to this, in some situations it may not be practical to use FLMs instead of TEMs. Examples may be situations where a third party code base with incorrect dependency information is integrated into a Symbian platform build.

Benefits

Short-term benefits

The following benefits are available in initial releases of RAPTOR.

Incremental builds (speed)

Because RAPTOR generates complete and correct dependency trees, RAPTOR will be able to support incremental builds. Incremental builds lead to improved build performance because, only a subset of files in the system need to be re-built.

Both developers and system integrators will benefit from incremental builds. We do however expect that incremental builds will be used mainly by developers.

Parallel builds (speed)

The key benefit of RAPTOR is in enabling fast parallel builds on different hardware platforms. The system is primarily designed to benefit system integrators (in other words, those using automated build systems), but will also benefit individual developers that can schedule builds on centrally-managed parallel build hardware through appropriate web interfaces, or interfaces in the IDE (neither are supplied by Symbian).

More flexibility

A key benefit of RAPTOR over SBSv1 is the flexibility achieved through powerful abstractions that enable extensibility in the following areas:

• The ability to modify properties of the build by extending build configurations (arguments that control the build) in different ways.
• The ability to extend the core of the build system by extending the FLM library.
• New ways to interact with the build system, such as the ability to address and build objects in the system model.

Both developers and system integrators will benefit from the extra flexibility that RAPTOR provides. Initial releases of RAPTOR provide significantly more flexibility than SBSv1. However, the need for backwards compatibility limits the flexibility a little, for example, it is not possible today to extend the build file formats, such as MMP files, or define your own formats. It should be noted however, RAPTOR’s architecture allows Symbian to implement such functionality in future should customers require such extensibility.

Mid-term benefits

In the future, Symbian can deliver additional benefits by building on top of the concepts that the initial release of RAPTOR introduces.

Integration into IDEs

RAPTOR is designed to allow easier integration of build functionality into IDEs. It does this by using the following architectural concepts:

• The RAPTOR API allows an Eclipse-based IDE to interact with RAPTOR. Large parts of RAPTOR are implemented in Python (and not in Java, as Eclipse is), which makes it possible to use Python implementations such as JPython or Jython, that seamlessly integrate with Java, and as a result, integrate with Eclipse1.
• The concept of RAPTOR build configurations (arguments that control the build) conceptually maps directly onto the concept of describing build settings in IDEs.
• Improved machine-readable logging formats that can be processed easily by IDEs.

Product-line engineering in production environments

Product-line engineering is a method that creates an underlying architecture of an organization’s product platform. It provides an architecture that is based on commonality and similarity. Different product variants can be derived from the basic product family, which creates the opportunity to reuse and differentiate products in the family. Currently, during mobile phone production, software product-line engineering takes several forms:

• The creation of software variants by compiling the same source code into different sets of variants.
• A strong component model that allows the assembly of a product from different components.
• The managment of software variants through different settings and other run-time configuration mechanisms. Note that most run-time configuration mechanisms can be seen as a generalized form of settings.

Symbian OS has supported functionality that enables variant creation in various forms for some time, particularly through strong feature and configuration management. The RAPTOR concepts of build configurations, variants, and filters in build specifications map to, and interact directly with existing concepts in Symbian OS. This means, with RAPTOR builds, it is possible to create and integrate systems that describe and manage software configurations centrally. Such systems would generate RAPTOR build configurations or variants automatically, which can then be applied to RAPTOR builds.

Managing settings is an area which is less established in Symbian OS, but is conceptually similar to build configurations. In a general sense, settings are no different to vectors of (parameter, value) pairs that are encoded in different data formats. This means that systems that manage settings for phone configurations are conceptually similar to systems that manage software configurations, which again are conceptually similar to build configurations. Settings management systems could generate settings files or repositories in different binary formats automatically, which in turn could be fed into RAPTOR through generated build configurations or build specifications. The latter can be achieved by using the RAPTOR API, which could be used to inject build artefacts into RAPTOR builds.

All file formats that RAPTOR has introduced are XML, in order to allow easy machine generation of data files such as build configurations.

Integrating other phone software production use-cases

The generic concepts of FLMs, FLM libraries, build specifications, and variants introduced by RAPTOR allow other tools that are part of the software production process to be migrated into the RAPTOR front-end. This allows the use of the hardware, and as a result, the use of parallelization capabilities that RAPTOR introduces for non-build use-cases that are important during phone software production. It also enables Symbian customers who use Electric Accelerator to make use of Electric Accelerator for these use-cases.

Examples of such integrations are as follows.

• Static analysis tools can be integrated by extending FLMs (or adding additional FLMs) and build configurations.
• Tools for checking properties of the code base, such as CDB and AURA, can be integrated by adding additional FLMs and build configurations to RAPTOR.
• ROM building can be supported by adding additional FLMs and build configurations to RAPTOR.
• Delivery tools, such as the CBR tools (in particular, the generation of CBR archives after a build), can be integrated into RAPTOR by adding additional FLMs and build configurations.

This requires that these tools are available on the host platform on which RAPTOR is run.

Migration to RAPTOR

Steps taken to ease migration

The following steps have been taken to make migration to RAPTOR as easy as possible.

Backwards compatibility

RAPTOR is compatible with the interfaces of SBSv1, bld.inf files and MMP files are retained, as are the bldmake and abld commands. These commands become proxies which pass the appropriate arguments to RAPTOR. It is expected that SBSv1 will be used alongside RAPTOR for approximately 12-18 months, with users gradually migrating to RAPTOR as they require increased performance and new features. Once RAPTOR is released, no new features will be added to SBSv1, although it will be maintained for up to 2 years.

Binary equivalence

We have also implemented binary equivalence in RAPTOR, meaning that building source code with SBSv1 and RAPTOR on any platform will result in the same end-product.

What needs to be done to migrate to RAPTOR

The following differences between RAPTOR and SBSv1 are likely to affect many of Symbian’s customers, and will need to be addressed before RAPTOR can be used.

Migrations for all customers

Build scripts

This section is applicable if you have build scripts that wrap SBSv1. This migration only applies for system builds, not for component builds. To make use of the full parallel build capability of RAPTOR it is necessary to use the system_definition.xml file which describes the components that are used in a build. Without this file, RAPTOR does not know which components constitute your system and thus cannot generate a makefile that contains a complete dependency tree. Note that RAPTOR only needs the <systemModel> section of the file. RAPTOR ignores the <build> part of the system definition which is used by a wrapper script available with SBSv1.

In order to migrate, modify the build scripts so that they use RAPTOR with the system_definition.xml file.

BR.1968 RAPTOR produces a different logging output

This section is applicable if you use tools which parse the output that SBSv1 produces. RAPTOR produces a different logging output from the current build system. The output has been designed to be easily parseable by tools such as an IDE, and to avoid corruptions of logs, as is often seen in distributed execution environments. The following example shows a small part of a RAPTOR log.

The logging output is supported by the Symbian log summary tools, meaning that tools based on these, using the summary output, will not be affected.

Any tools that rely on the log output that have been developed outside of Symbian may need to be modified to support the new log format.

Missing dependencies

This section is applicable if your build breaks because of missing dependencies that RAPTOR cannot deduce. In some rare circumstances RAPTOR cannot automatically deduce dependencies between individual files, for example header files that are generated during resource creation and used by source files.

In Symbian’s code base there have been approximately 50 instances of this problem, where one generated resource header file is used in 50 source files.

In order to migrate, dependencies in MMP files will need to be specified using the DEPEND keyword. This keyword has also been added to SBSv1. For convenience, a global list of dependencies has been introduced to avoid having to change individual MMP files.

Migrations that apply to customers that use legacy SBSv1 functionality

Extension makefiles

This section is applicable if you are using GNU or NMake extension makefiles.

RAPTOR does not support GNU or NMake extension makefiles. These have to be ported to TEMs or FLMs The reason for not supporting these extension types is that TEMs have been introduced to promote more re-use of build patterns in build system extensions, which is something that GNU and NMake extension makefiles do not provide. GNU and NMake extension makefiles encourage the user to copy and paste (resulting in long-term maintenance issues), while TEMs encourage code re-use in build extensions.

For information on migration, run a search on “Symbian OS v9.3 new build system features” in the Symbian Developer Library, and see the RAPTOR documentation.

ABIv2

This section is applicable if you are building your code base in ABIv1 mode. RAPTOR does not support compiling code in ABIv1 mode. Further note that future compiler versions, RVCT v3.x for example, do not support compilation in ABIv1 mode.

To switch to ABIv2 mode, edit the variant.cfg located at epoc32\tools\variant and add the macro ENABLE_ABIV2_MODE. For more information, run a search on “How to switch to ABIv2 mode” in the Symbian Developer Library.

Old build platforms

This section is applicable if you depend on legacy build platforms and want to use RAPTOR.

RAPTOR does not support some target platforms that SBSv1 supported. Symbian has no plans to support the following legacy target platforms:

• ARMv4, ARMv4T, VS6, VS2003, WINS, X86, Thumb, ARMvI.
• Note that the first released version of RAPTOR does not support ARMv5 ABIv1 builds, but support will be added in the future.

External code using these target platforms will not build without creating additional build configurations, or in some cases additional FLMs for RAPTOR.

Licensees will be able to create their own target platform configurations and FLMs using documentation provided with RAPTOR.

BR.1967 RAPTOR does not use BSF files

This section applies if you are using BSF files in SBSv1.

The current build system allows BSF files to be placed in /epoc32/tools to create new target platforms, by extending other target platforms. The new build system will not support this mechanism, so third-party tools requiring extra platforms in BSF files will break. In order to help with migration, a tool will be provided to translate existing BSF files into a build specification that is required by RAPTOR. However, until a tool is provided it is possible to define build specifications manually. The following example shows a build specification for an ARMv6 build.

 
<?xml version="1.0" encoding="ISO-8859-1"?>
 
<build xmlns="http://symbian.com/xml/build"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://symbian.com/xml/build 
http://symbian.com/xml/build/1_0.xsd">
 
 <!-- build configurations for ARM compilers -->
 
 <config name="armv6_udeb" extends="armv5_udeb">
  <var>
   <!-- link to bld.inf and mmp platform names -->
   <set name='TRADITIONAL_PLATFORM' value='ARMV6' />
 
   <set name='FULLVARIANTPATH' value='armv6/udeb'/>
   <set name='TARGET_ARCH_OPTION' value='--cpu 6T'/>
   <set name='VARIANTPLATFORM' value='armv6'/>
   </var>
 </config>
 
 <config name="armv6_urel" extends="armv5_urel">
   <var>
    <!-- link to bld.inf and mmp platform names -->
    <set name='TRADITIONAL_PLATFORM' value='ARMV6' />
 
    <set name='FULLVARIANTPATH' value='armv6/urel'/>
    <set name='TARGET_ARCH_OPTION' value='--cpu 6T'/>
    <set name='VARIANTPLATFORM' value='armv6'/>
   </var>
 </config>
</build>
 

Migrations that apply to Linux only

Preparing the code base for Linux

This section is applicable if you want to run RAPTOR on Linux on your own code base.

To perform builds on Linux, it is necessary to ensure that the code base compiled on Linux is ready for Linux. Symbian has made its own code base Linux-ready from Symbian OS v9.5 onwards and is planning to make Symbian OS v9.4 Linux-ready in the near future. The following two factors determine the readiness:

• The accurate spelling of filenames in the file system, making sure that they match the spelling of references to those files in source files. For example, the spelling of filenames in include statements needs to match the spelling of the file in the file system.
• The type of slashes that are used when referencing source files. This is important because the C and C++ standard states that only source code that uses Linux type slashes (such as like/this) is guaranteed to compile on all platforms.

For more information on the requirements for Linux readiness see the Filename Policy Migration Guide in the Symbian Developer Library.

SBSv1 has some options, checksource and fixsource, which are described in the Filename Policy Migration Guide. These options help to identify most of the violations of the filename policy, with the option to fix the code automatically. The remaining violations can only be fixed by iteratively building the code on Linux, and removing the remaining failures with each test build. Note that RAPTOR itself does some normalization of slashes where it can, for example, in MMP files.

Preparing Template Extension Makefiles for Linux builds

This section is applicable if you use TEMs and want to run RAPTOR on Linux.

RAPTOR supports TEMs, but TEMs in SBSv1 are not written to be platform independent. Most existing TEMs contain commands for the Windows command shell (CMD) and they need to be converted to become platform independent.

In order to help with migration, two make libraries are currently being developed that abstract Windows specific commands which exist in Symbian’s TEMs. All Symbian-owned tools that are part of the main build process have been ported to Linux15. A guide will be provided to help make TEMs host-OS independent, but basically, the following would need to be done:

• Add a standard preamble, which includes the correct make library, to the beginning of TEMs.
• Replace all direct calls to the command shell with system-independent make macros.

In addition, there may be some complications which will require extra work:

• If you are using tools that only exist on Windows, you will need to port these to Linux.
• If you are using tools that exist on Windows and Linux, but are not abstracted by Symbian’s make libraries, you will need to extend the abstractions provided by Symbian.

Migrations for optimization purposes

Replace TEMs with FLMs=

This section is applicable if you use TEMs that perform a significant proportion of a build from within the TEM, performance on a parallel build machine is not as high as expected, or there are correctness issues with incremental builds.

Although RAPTOR supports TEMs, all build steps in the TEM are treated as one build step, because a nested call to make needs to be performed. This means that RAPTOR does not see whole parts of the dependency tree, which can lead to computing resources on a parallel machine not being used. It can also lead to an incomplete dependency tree, which can cause incorrect incremental builds.

In order to help with migration, a mechanism has been added which allows the definition of FLM interfaces for individual TEMs. RAPTOR will automatically use the FLM instead of TEM. This approach allows RAPTOR and SBSv1 to be run on the same code base without modification. An FLM body and interface will have to be written for each TEM that is to be replaced by an FLM. Initially, this would only be done to increase the build system’s efficiency, but Symbian may deprecate support for TEMs in future releases.

The following example shows how this approach works. The sample below shows an invocation of a TEM.

 
…
start extension base\config
 
Option 	    PREFIX      _SH2_ 
option      HALPATH     \..\..\.. 
option      SOURCE      \..\hal 
 
end
…
 
 

The following FLM interface definition would ensure that RAPTOR re-directs to an FLM, rather than using the TEM.

 
…
<interface name="base.config" flm="base_config.flm">
	<param name='PREFIX' />
	<param name='HALPATH' />
	<param name='SOURCE' />
</interface> 
 
…
 

Migrations that affect customers who use internal SBSv1 interfaces

BR.1964 RAPTOR produces different makefiles

This section is applicable if you use scripts to modify the makefiles that SBSv1 has created when running a set of builds, rather than changing parameters, and regenerating the makefiles using SBSv1.

RAPTOR produces makefiles that are different from those produced by SBSv1. If you have scripts or tools that modify makefiles that SBSv1 generates, these scripts will break with RAPTOR.

In order to migrate, always use the extensibility features of RAPTOR to generate new makefiles.

BR.1966 RAPTOR will not include the makmake tool

This section applies if you use scripts that call the undocumented makmake tool directly.

RAPTOR will provide backwards compatibility with SBSv1, via a wrapper script abld, which will drive RAPTOR based on the same options as currently supported by the abld command. There will be no wrapper script for makmake, so any third-party tools which drive a build using makmake will not work with RAPTOR.

In order to migrate, always use the extensibility features of RAPTOR.

Comments

Bdonegan said…

Larsk said…

Tw 2 said…

Sign in to comment…