Playability Testing In Symbian

What is Playability Testing?

There are two main groups of playability issues:

Integration: getting all the software components to play nicely together, for example in the way data buffers are handed between components, and the synchronisation surrounding this.
Hardening: ensuring those components can handle every stream type. Often encoders and muxers have different interpretations of standards that means some players can't play certain streams. Or streams that are compliant have certain characteristics that form corner cases, such as streams that push the envelope in the "acceleration" of data consumption.

Symbian hosts a database of use-case level playability tests

Test = use-case description + content item + reference platform
Symbian hosts copyright-free or contributed content
For copyrighted content, database references the content’s owner(s) – thus allowing integration with members’ own test infra

Use-case level testing is the focus since this exercises testing at a systems level.

User logs in and sees a prompt asking how they would like to contribute – contribute testing time, contribute test definitions
If they contribute testing time, they see a set of test suggestions – the top “N” least-recently-run tests in both failed tests and passing tests (for regression testing), and a list of items awaiting reproduction. The reproduction list is sorted by bugzilla ID (instead of Test ID), since it is the bug that needs reproducing and there may be multiple bugs associated with a given test case, some reproduced and some not
All activities on a test are logged with date/time, user ID and action – this allows track-back if mistakes creep in
All actions require corroboration by another tester (new tests require approval, newly discovered bugs require reproduction, newly closed bugs require double-checking)

Scale: need to run tests often and frequently, which is a significant overhead if each company approaches individually. This applies even to passing tests, since changes can easily cause regression
Commonality: between the kinds of tests different members will run, and hence openness can avoid wheel-reinvention
Diversity of tests: more perspectives on playability testing mean a broader variety of defined tests
Easier Product Creation: resolving issues in a coordinated fashion at the platform level makes targeting bugs during product creation much more efficient
Better platform for service providers: a broad effort to improve multimedia QA will make Symbian devices more appealing to service providers