Quality Metrics
From Symbian Developer Community
This page is here as a focus for material relating to Quality Metrics, including those that Symbian Foundation is considering as candidates.
Metrics Archives
First milestone Metrics have now been rolled out. Please visit the Archives Page.
Initial Candidate Metrics
This document describes some Quality-related metrics that Symbian Foundation is considering for implementation.
This page shows some Quality-related metrics initially requested by Symbian Foundation members.
Metrics dashboard proposed (with example numbers)
Agreed list of Metrics for the first 3 months
As on 1st July 09, an initial list of metrics was finalised and a status dashboard has been kick started and maintained to indicate the status and progress of the metric implementation process. Since individual metric item implementation belongs to different owners and the complexity of implementation being varied, a dashboard is presented here to help understand the current status and progress made in each of them.
The actual Metric implementation may happen through a few stages before they are functionally complete and achieve what is intended. For a detailed view on the QWG status dashboard read here.
As a first step towards implementing the Metric, participants may fill out a 3 slide powerpoint presentation file that helps define the details of a Metric in Metrics Template. The filled out file may then be circulated amongst the members (and also appropriately shared with the community) for reviews.
If you would like a .ppt version of the empty template, please send an email to nithyar(at)symbian.org
Metric definitions
For a detailed explanation on various Metric definitions, please read here
Metric implementation status dashboard
| Metric | Current Status | Expected date of progression to next stage | Assigned to | Update so far |
|---|---|---|---|---|
| Defect Inflow | Implemented | 10/08/09 | TW | |
| Defect Outflow | Implemented | 10/08/09 | TW | |
| Open Defects | Implemented | 10/08/09 | TW | |
Turn Around Time
| Implemented | tba | AH | |
| Package Size | Implemented | tba | RB |
|
| Code Coverage by the Symbian Foundation | Initial | tba | SF | |
| Unplanned BC breaks per package per year | Initial | tba | VP/Build Team | |
| Build Success Rate | Defined | tba | VP/Build Team | |
| Code Churn | Implemented | tba | RB |
|
| ROM Usage per package per build | Initial | tba | tba | |
| Defect Introduction Rate | Initial | Metric to be considered for second iteration | tba |
Monitoring development by measuring: Questions and responses
Is the package community diverse?
This would need a metric based on data on submissions to a package code-base. We don't know what data SF will be collecting on a submission-by-submission basis in the future, but obviously we need to hold data on submission-contributor to support a metric like this.
Do they fix critical defects quickly?
This would be based on the Fix Turnaround Time metric. We would need to ensure that the dates stored/displayed in Bugzilla (e.g. for New Defect, Verified:Fixed) are essentially the same as those from the original system from where the details were cross-posted (e.g. TeamTrack, Trinity).
Do they accept external contributions?
A tricky one. There might be a situation where no external contributions were made, so none could be accepted. Do we want a metric like: the proportion of external contributions made that were accepted? Or do we want something simpler, like: Number of external contributions made? We also need to decide whether or not the count would make a distinction between defect fix contributions and functional additions. There is an assumption that we would collect and hold data on submission/contribution acceptance.
If, by external contributions, we really mean fixes then we would want to collect and record data in Bugzilla that would allow us to distinguish between fixes supplied by (a representative of) the package owner, and fixes supplied by external contributors.
Do they prioritise quality over content?
Again, a tricky one. You might think that the simple Defect Introduction Rate would be the straightforward metric here. However, if the only submissions are for defect fixing, you would get a relatively high DIR figure (indicating lower quality) whereas defect fixing might be thought of as prioritising quality over content. So we need to use this metric carefully, and look at other indicators, e.g. test effort (if we have it), Relative Defect Inflow and Trend of Defect Inflow.
Is the new Package version being hardened, yet?
The Defect Inflow and Defect Outflow metrics come to mind here, particularly the Trend of Defect Inflow, Trend of Defect Outflow. However, we would need to confirm levels of test effort*.
- It is just possible that, in the “new world” of SF, we will be able to assume that test levels, at least those of external contributors, are always nearly constant. This is because there is a relatively large body of people worldwide who have constant access to the code, and who want to do stuff with it. All we need then is a marker in Bugzilla that allows us to distinguish between those defects raised by (a representative of) the package owner, and those raised by external contributors.
Are there still many defects to be found in the Package version?
The Estimated Defect Density or Estimated Number of Unresolved Defects (ENUD) metric comes to mind here. Again, we would need to confirm levels of test effort.
Other Questions
- How much visibility is given of changes to the Package?
- How much is planned vs. a surprise?
- How often is the Package updated?
These questions rely on metrics that are based on Requirements and Submissions data. We have not finalised the data categories that SF plans to collect and hold in these areas. However there are a number of old Metrics that Symbian SW used to gather which might be of interest:
- Next External Release Content Predictability
- Unapproved Interface Breaks
- Unplanned Deliveries
- Release Schedule Churn
Work Group Meetings
- Kick-off - 2 June 2009
- Face-Face - 17 June 2009
- QWG Conf. call - 01 July 2009
- QWG Conf. call - 23 July 2009
- QWG Conf. call - 17 August 2009
- QWG Conf. call - 26 August 2009
- QWG Conf. call - 03 September 2009
- QWG Conf. call - 10 September 2009
- QWG Conf. call - 17 September 2009
- QWG Conf. call - 07 October 2009
- QWG Conf. call - 13 October 2009
- QWG Conf. call - 12 November 2009
Participants
- Stefan Fager, Fujitsu (SF)
- Pierre Leblond, ST Ericsson (PL)
- Tom Whitson, Accenture (TW)
- Laura Martin, Accenture (LM)
- Andrew Hodgson, Nokia (AH)
- Victor Palau, Symbian Foundation (VP)
- Richard Baruch, Symbian Foundation (RB)
- Nithya Rajendrababu, Symbian Foundation (NR)
Mailing List Information
Sign in to comment…
