Symbian developer community

 
Home > Wiki > Test Working Group
wiki

Test Working Group

From Symbian Developer Community

Jump to: navigation, search
Copy editing required
This article has not yet been copy edited


Contents

What is the Test Working Group (aka "Test WG" or "TWG")?

Give explanation of what is the Test WG

It is great to be able to bring together key expertise from our member companies to discuss the test activities related to the Symbian Platform and how these can be realised.

Guidelines for WGs can be found here: http://developer.symbian.org/wiki/index.php/Working_Group_Guidelines

Back to top

Why Test WG has been created?

Give the reasons behind the creation of the Test WG

This is a topic that has been raised in the SF Release Council as well as other forums and clearly is of great interest to anyone adopting Symbian Releases to create devices, adaptations, applications or services.

Back to top

What are the objectives the Test WG?

Objectives / Jim's doc (link)

Symbian Test Approach v1 Nov09

Back to top

Who is part of the Test WG?

Contacts list: TWG_Contacts

Give an introduction of the different people

Janne Kolari (JK) - Digia, Finland

Tom Whitson (TW) – Docomo/Accenture, UK

Chandra Narayanasamy (CN) - Fujitsu/Cell, UK

Andrew Cooper (AC) - Nokia, UK

Mikael Christersson (MC) - SEMC, Japan

Pierre Lablond (PL) - STE, France

Fred Maurel (FM) - TI, USA

Jim Clarke (JC) - Symbian, UK

Mark Skrebels (MS) - Symbian, UK

Hi, I work in Technology & Delivery Management in the SF. My roles are:

  • Senior Release Manager (looking after the S^n release programme)
  • Release Council Chair

I am also Delivery Management's interface to our Japan Alliance team.

My previous experience in the mobile industry was with Symbian Ltd (which was acquired by Nokia at the end of 2008) as Technology Programme Manager in the Software Engineering organisation.

Arnaud Lenoir (AL) - Symbian, UK

Mike Kinghan (MK) - Symbian, UK

Back to top

Test WG organisation

This describe who is the chair of the Test WG, take minutes, in charge of Wiki

Back to top

Tests

We are here to do work around tests, then we need a section to complete

Tests framework

ATS, ...

What are the different type of tests?

Include a diagram showing the interaction between the different tests (smoke, sanity, system, ...) Also include a diagram showing when the different tests are run in the like of a package, platform, kits, ...

The following link describes the interaction between the different type of tests and when they are done: [Continuous Validation Cycle]

How to make sure that my package contain all the tests we need?

What we are expecting to be delivered?

Each package needs to have tests supported in different format like stif, tef, RTest, ...

Template

Give some examples how files need to be organised, what to include, ... This is to make sure that if a package owner wants to make is package compliant for the point of view of tests, he knows exactly what needs to be done and it's straight forward. Also indicates that new packages contributed should have that too from the beginning (best) or to have it at some point (need chasing). 1st option probably best as, the work is done from the beginning and just need to fill the gaps.

Back to top

Tests gap analysis

This section should be a link to the the document done by Arnaud showing what we have and what is missing

Analysis

Statistics, ...

For the TWG meeting on 15/12/2009, a first basic analysis (as the script used was not fully finished and the list of things to have a look at was not fully understood and complete) was done of the presence of tests in all the packages. The results presented at that meeting could be found here: 1st package test status analysis.

Now that the script used to detect tests patterns in the different packages and the list of patterns to have a look for is now understood and complete, I have ran the script to get a more appropriate results that reflect more the reality of what we have today.

To do the comparison between S^2 and S^3, I used respectively PDK_2.0.1 & PDK_3.0.d as it's what I have been using to do my tests and it's no too far away from the latest releases for S^2 & S^3.

Of course, in due time, we will be able to run these tests regularly on the latest releases to compare the improvements in the tests between releases. We will include the scripts as part of the FBF build tool.

Here is the file generated: Full package tests dection analysis 15-01-2010

Note that the values indicated are showing the presence of tests of any kind. No tests have been ran following the use of that script and the values given are not representatives of the goodness of a package. The test results per package will come soon as it is part of the ongoing development around tests.

Priorities

1st job to be done KERNEL. It's not what the vote said! Is it ok to have kernelhwsrv with a lower priority? That is the question!?


We have asked 4 people inside Symbian Foundation, who have different backgrounds and points of view to vote on which packages we should concentrate first in terms of testing and we had a contribution from a 5th person from Accenture who was representing the Japanese market.

To give you an idea of who voted, here are their roles:

  1. 1st person: Chief Integration Engineer
  2. 2nd person: Technology Specialist
  3. 3rd person: Test Lead
  4. 4th person: Technical Lead
  5. 5th person: External contribution (Accenture / Japanese market)

The voting criteria were:

  1. Vital package for the Symbian platform (eg?: kernelhwsrv)
  2. Vital package for S^3 (eg?: UI)
  3. Important functionality for handset vendors to create products
  4. Allow for a mix of test frameworks (eg: TEF, STIF and RTEST tests)


  • Each person could put a note between 0 and 10 for each package and there were no limits in the number of the same notes given to packages (could give several 10 for example)
  • 10 being "high"
  • 5 being "medium"
  • 0 being "low"
  • The classification was done by adding the total of the votes for each package and sorting out the packages by "weight" from the biggest numbers to the smallest numbers


Concerning the prioritisation, we wanted to have between 5 to 10 packages to start with. Now the vote gave a clear winner but there are packages that ended up with the same "weight".

1. Do we take the 1st 10 packages listed even if for example the 10th package as the same weight as package 11th and 12th

OR

2. Do we classify packages with the same "weight" with the same ranking and take the packages ranked from 1 to 10th?

I decided to take solution 2, which give us more packages than 5 or 10 depending the limit we take.

Now this is what I have decided to get a starting point to allow us to have a discussion and as a group to get an agreement on what the exact number we should concentrate on should be. Please let others know your comments using the mailing list email.

Everybody seems happy with the solution 2 and therefore it's what we will use.

It was decided to split the work into 3 phases.

  1. 1st phase: the packages ranked from 1 to 10
  2. 2nd phase: the packages ranked from 11 to 20
  3. 3rd phase: the remaining packages ranked from 21

You can find the results in the column "Priority (new)" (column J) in the Excel spreadsheet link further down this section. Colour scheme associated to the different phases:

1st phase highlighted in yellow
2nd phase highlighted in light blue

Based on that,

  • The 10 first ranked packages (1 to 10) for the 1st phase give us a total of 12 packages:
  1. \sf\os\graphics
  2. \sf\app\homescreen
  3. \sf\app\organizer
  4. \sf\adaptation\qemu
  5. \sf\mw\appinstall
  6. \sf\os\commsfw
  7. \sf\app\musicplayer
  8. \sf\adaptation\beagleboard
  9. \sf\os\kernelhwsrv
  10. \sf\os\mm
  11. \sf\mw\uiaccelerator
  12. \sf\app\radio


  • The next 10 ranked packages (11 to 20) for the 2nd phase give us a total of 26 packages:
  1. \sf\os\textandloc
  2. \sf\mw\btservices
  3. \sf\app\contacts
  4. \sf\mw\classicui
  5. \sf\mw\mmappfw
  6. \sf\mw\mmmw
  7. \sf\mw\mmuifw
  8. \sf\os\cellularsrv
  9. \sf\os\networkingsrv
  10. \sf\os\persistentdata
  11. \sf\os\security
  12. \sf\app\graphicsuis
  13. \sf\app\photos
  14. \sf\os\bt
  15. \sf\app\phone
  16. \sf\app\videocenter
  17. \sf\mw\imghandling
  18. \sf\os\boardsupport
  19. \sf\mw\homescreensrv
  20. \sf\app\commonemail
  21. \sf\app\devicecontrol
  22. \sf\mw\uiresources
  23. \sf\app\files
  24. \sf\mw\ipconnmgmt
  25. \sf\mw\locationsrv
  26. \sf\mw\securitysrv

For the 3rd phase, please consult the link below to the Excel spreadsheet to find out the pacakges that have no colour scheme associated with.

You can find the document containing the detail of the votes here: Packages_tests_prioritisation.xlsx

Actions to close the gaps

  1. The next step is to start to work on the 1st phase for the packages priority. This work will consist of contacting the package owners for these 12 packages that are included in that 1st phase for which we will establish the test status for each package with the following set of questions (non exhaustive):


  1. Who is the person in charge of the tests for a package (maybe the package owner or maybe someone else?)
  2. Are they happy with the tests coverage they have currently?
  3. Do they think they need to improve the tests coverage?
  4. Are they expecting to add new tests for new features related to the package? If yes, what sort of figures are they looking at in terms of effort, date, ...?
  5. Do they think they need to re-organise the tests (architecture, move to a different type of tests, ...)?
  6. Can their tests to be automated or not? If not what is the percentage that is automated.
  7. Amongst all the tests they have for a package, do they have short list of these tests to run some sanity tests, smoke tests and regression tests (Here is the page describing the different type of tests: http://developer.symbian.org/wiki/index.php/Continuous_Validation_Cycle)?
  8. Do they have some tests results that they can share (total, pass rate, NA, ...)
    1. With us (restricted to some people only in Symbian? TWG?)?
    2. With the community (public)?
  9. Anything else we think that could be appropriate?


This action will be done by AL with help from AC.

Follow-up

Things done and finished for the point of view of the Test WG but some actions are required from external teams

Back to top

Getting tests results for top 10 priority packages

Now that the analysis phase & priority list established, it's time to get some concrete work done and get some tests results for the top 10 priority packages.

You can follow all the work done around these top 10 priority packages on the following page: "Getting tests results for top 10 priority packages" plus the story to finish the remaining work for all other packages.

Back to top

Actions

Back to top

Test WG Blog

This is the link to the Blog for the Test WG. Do we need one?????

Back to top

FAQ

What do I need to do to join the Test WG?

This section is about what to do to be part of the Test WG

How can I contribute to the Test WG?

Explain what we are looking for in terms of contributions and how people can contribute

Back to top

Meeting history

Back to top

How to contact us?

If ever you wish to contact the Test WG, a maillist has been created for that: test-wg@lists.symbian.org

You can subscribe to the mailing list from this link: Subscribe to TWG mailing list

Back to top

Useful links

In this section you will find interesting links related to the Symbian Foundation tests that you can use to improve your knowledge or as reference.

Testing Guidelines for Package Releases

Symbian Test Tools

Syborg - QEMU

Binary Compatibility Suite Quick Start

Back to top

Retrieved from "http://developer.symbian.org/wiki/index.php/Test_Working_Group"

Comments

Sign in to comment…