Symbian developer community

 
Home > Wiki > Symbian Signed Test Criteria V4 Wiki version
wiki

Symbian Signed Test Criteria V4 Wiki version

From Symbian Developer Community

Jump to: navigation, search

Contents

Introduction

Background

These tests will be used by the Symbian Signed Test House to conduct the Symbian Signed testing for both Certified Signed and for the audit of an Express Signed submission.

They must also be used by developers to ensure that their application satisfies the criteria for Symbian Signed prior to submission for either Express Signed or Certified Signed.

This version of the test criteria is in effect from 5th January 2010.

A print-ready PDF version of this document is available for download here

There is also a Simplified Chinese PDF version available for download

In addition to the versions above, members of the Symbian Community have contributed the following translations of the test criteria:

What is included in the testing — and what is not

The tests contained in this document have been mandated by Symbian, in agreement with key partners in the community, as the minimum set of criteria for an application to be Symbian Signed.

We explicitly exclude any evaluation of the following from our test criteria:

  • Universal device coverage — submissions are tested on representative devices and not every device
  • Application functionality — we make no judgement on the functionality of the application as long as the test criteria are adhered to
  • Adherence to any published style guides

Finally, Symbian Signed does not evaluate the content of the application in any way and no judgement is made as part of the Symbian Signed process on the appropriateness of the content of any submission.

Providing Guidance to the Test House

Due to the nature of some of the tests and the diversity of mobile applications, some guidance from the developer may be necessary to explain how certain test cases apply to their application.

The notes accompanying each test case explain where guidance may be necessary.

Any such guidance should be given by the developer in the submission statement.

Testing VoIP applications

If you are testing a VoIP application, then there are some special tests which must be performed due to the nature of the application. Details can be found in the section "Additional Tests for VoIP applications".

Test Criteria

Submission Checks

CHECK 1 — Valid PublisherID
The submission must be signed with a valid PublisherID.

Submissions will be rejected if they are not signed with a Publisher ID or if the Publisher ID they are signed with has been revoked or is otherwise invalid


CHECK 2 — Correct UIDs
Application UIDs

The application UIDs must be from the protected UID range, and the UIDs must be allocated to the Symbian Signed account being used to submit the application.

Vendor ID

If a VID is specified, it must come from the correct UID range (0x70000000-0x7fffffff) and must correspond to the company submitting the application.

If no VID is specified, then a value of 0 should be included.

Package File UID

The package UIDs specified must be owned by the developer making the submission. For Symbian v9.x and later the package UIDs must come from the protected UID range and the UIDs must be allocated to the Symbian Signed account being used to submit the application.

If the package file UID is 0x10202BE9 (update to central repository) or 0x101F7989 (update to c32) then you must work with a device manufacturer or Symbian to get your submission signed.


CHECK 3 — Completed Submission Statement
The submission statement must be completed with all required information.


Symbian Signed Tests

TEST 1 — Installation
TEST STEPS
Before starting the test round, use a file manager to note the free user space available on the phone. You will need this information in test 8.
1 Install the application being tested.

The application must install without error.

2 During installation note the version number presented to the user.

The version number must match that specified during submission.

3 Verify that the application has successfully installed on the device by navigating to the area on the phone where new applications are installed.

The application should present one or more icon(s) on the phone.

Notes

For any submissions which do not appear obviously once installed, the submitter must include details in the submission statement of how successful installation can be verified.

If the content does not appear obviously on the device once installed, and specific instructions are lacking in the submission statement, then this test will be failed.


TEST 2 - Application start/stop behaviour
TEST STEPS
1 Start the application by selecting the icon or following the steps outlined in the submission statement

Navigate to the Task Manager and check that the application appears there.

2 Close the application from the Task Manager.

Exit the Task Manager, and re-launch the Task Manager.

The application must no longer appear in the Task Manager.

3 Start the application as in Step 1.

Go to the Task Manager to verify that the application is running.

The application must appear in the task manager.

4 Close the application from within the application UI and then return to the Task Manager.

The application must no longer be running and must no longer appear in the task manager.

5 Restart the application as in Step 1.

Navigate to the Task Manager.

The application must once again appear in the Task Manager.

Notes

An application which must run in the background does not need to appear in the Task Manager or present a UI so long as the developer justifies this behaviour during submission.

All applications must have some way of verifying that they are running on the device, though, and the developer should provide this information.


TEST 3 — Application credentials
TEST STEPS
1 With the application running, check the name of the application displayed on the phone.

The application must display the same name on the phone as stated during submission.

2 Note the functionality of the application as it runs on the device.

The basic functionality of the application must match that declared during submission.

Notes

Step 1 does not apply to applications which do not have a UI

VoIP applications must present a UI in order to pass this test.



TEST 4 — No disruption to voice calls
TEST STEPS
1 With the application installed and running use a second phone to call the test device.

The incoming call must be indicated to the user on the test device.

2 Answer the call on the test device.

You must be able to conduct a conversation with the other party without interference from the application being tested.

3 End the call in the normal way on the test device.

The voice call must be ended.

4 From the test device, make a call to a second phone. Answer the call from the other device.

The call must be indicated on both devices, and you must be able to conduct a conversation with the other party without interference from the application being tested.

5 End the voice call from the second device.

The call must be ended on both devices.

6 Place a test call to the emergency 112 number from the device.

*Please check in your territory for the approved way to make test calls to the emergency services.

Notes

If the application being tested has the MultimediaDD capability, and has audio functionality, then that functionality must be in use whilst this test is performed. Particularly, it should be checked that the audio from the application is faded down to allow the user to hear the telephone call.

VoIP applications will need this test running using both the handset held to the user's ear and using a headset. The test should be run with a VoIP call in progress, and the incoming GSM call should be announced with call waiting tones.


TEST 5 — No disruption to text messages
TEST STEPS
1 With the application installed and running, send a text message to the test device.

The incoming text message must be notified to the user as per their alert settings.

2 Read the text message on the test device and choose to reply. Send the reply.

The reply must be received at the second device.

3 From the standby screen on the test device, navigate to the "new text message" option and create a new message. Send the message to the second device.

The message must be received at the second device.


TEST 6 — Auto-start behaviour
TEST STEPS
1 With the application running, find the settings for the application — either within the application itself or from the settings option on the device.

There must be an option which allows the user to enable/disable auto-start functionality.

2 Ensure that the setting for auto-start behaviour is disabled, and restart the device.

The application must not start on device boot.

3 Now change the setting so that auto-start behaviour is enabled for the application and restart the phone.

The application must start when the phone boots.

Notes

If the application does not have auto-start functionality, then this test does not need to be run.



TEST 7- No disruption to key device applications
TEST STEPS
1 Ensure that the contacts, messaging and calendar applications are populated with data and start the application as in Test 2.

After the application has been installed and used, the data entered into those applications must not be altered in any way without the user being aware.

2 With the application running, navigate to the messages application and create a new message.

Save that message to the drafts folder and then open and edit it.

Finally, delete the message from the drafts folder and delete a message from the inbox.

All of the above actions should be possible without interference from the installed application.

3 Navigate to the contacts application.

Create a contact, then edit that contact and then delete it.

The application should not interfere with any of the actions above without notifying the user and giving them option to avoid the change.

4 Navigate to the calendar application.

Create an appointment in the calendar. Edit the appointment and then delete it.

The application should not interfere with any of the actions above without notifying the user and giving them option to avoid the change.

5 Use the web browser on the device to go to a web page which is known to work on the network being used.

It must be possible to create a data connection and to access the web page selected.

Notes

If the application, as part of stated functionality, makes changes to user data then an exception can be claimed here. The functionality must be described in the documentation with the application and all data other than that mentioned in the user guide must remain untouched as described in the test case.

The data used in this test case is also needed for Test 8, so leave the data on the device when proceeding straight into Test 8.



TEST 8 — Un-install
TEST STEPS
1 Stop the application as described in Test 2 and uninstall the application using the system installer.

The application must be uninstalled without error.

2 Following the same steps as in Test 1, navigate to where you would expect to see the application icon.

The application icon must not longer be present on the device.

If you used another method to verify successful installation in Test 1 then use this method to ensure that the application has been uninstalled.

3 Check the contacts, messages and calendar applications to ensure that that the data present in Test 7 is still present in those applications.
4 Using the same file manager as at the start of Test 1 check that the amount of user space available on the device is either the same as that found in step 1 or that any difference between the space available before and after fulfils the following criteria.

a) Excluding user-generated and downloaded content, the application leaves no more than 100Kb of data on the phone after uninstall

b) Any data left on the device after install matches the explanation given during the submission process

Notes

You should start this test with the application data from Test 7 still in place on the device.



TEST 9 — Device adaptation
TEST STEPS
Note: The following test steps should be run on the list of devices corresponding to the UIDs specified in the .pkg file.

The lead device list can be found at http://tiny.symbian.org/devicetable

1 Install the application onto the device

The application should install on the device or present an error message to explain that it cannot install onto that device.

2 Launch the application.

The application should run on the device or present an error message to explain that it cannot run on that device.

3 Briefly examine the application whilst running.

UI elements should be functional and text should be readable in the main screen of the application.

4 If the device on which the application is currently being tested supports portrait and landscape screen modes, start the application and then switch between the screen modes.

The application should continue to be functional, and usable, in both screen orientations of the device, whether or not the application rotates in response to the screen mode change.

5 Close the application from the application UI

The application should stop running.

6 Uninstall the application from the phone.

The un-installation should happen without error and the application must be un-installed.

Notes

Applications which do not present a UI to the user in normal usage do not need to run this test.

On the primary device — on which all of the other test cases have been run - only step 4 of this test should be performed as all of the other steps of this test case are covered elsewhere.

Additional Tests for VoIP applications

Note that Test 3 and Test 4 both contain additional notes which apply to the testing of VoIP applications. Please read and apply these notes when running those tests on VoIP applications.

Test 10 — Additional emergency call testing for VoIP apps
TEST STEPS
Note: These test steps should be performed twice — once with a SIM card in the device and once without.
1 With the VoIP application running in the background, but with no VoIP call in progress, initiate an emergency call in the usual way.

The emergency call must be placed over the GSM/CDMA network successfully.

2 With the VoIP application running in the background with a VoIP call in progress, initiate an emergency call in the usual way.

The emergency call must be placed over the GSM/CDMA network successfully and the VoIP call should be terminated or placed on hold.

3 With the VoIP application in the background, and an emergency call active make a VoIP call to the device.

The incoming VoIP must be rejected, and the emergency call must not be interrupted.

Appendix A: Waivers

What is a waiver?

If your application doesn't meet one (or more) of the tests, then you must apply for a waiver. A waiver is a statement from Symbian — agreed by the OEMs — that it is acceptable that your application does not meet a certain test case and that your application can be signed even with that omission.

Do I need a waiver?

If your application doesn't fulfill a particular test case, then you should first check the notes underneath the test as it the test may not apply to your particular application.

If your application is not covered by the notes, then you must submit a waiver request and have it approved before your application will be signed.

If you are using Express Signed, then you must submit your waiver request before making your submission as waivers will not be retrospectively approved for failed audits.

How do I apply for a waiver?

Details of how to apply for a waiver can be found at http://developer.symbian.org.

Decisions on waivers will be made as quickly as possible, and you will be informed as soon as a decision has been made.

Appendix B: Device Manufacturer Capabilities

What are device manufacturer capabilities?

The three most sensitive capabilities — AllFiles, DRM and TCB — are available to developers only when working in cooperation with a device manufacturer. They are not available through the normal Symbian Signed channels.

How do I get permission to use those capabilities?

You must approach the device manufacturer with whom you have a business relationship and seek their permission to use these capabilities. If you do not have existing contact details for the manufacturer, then please contact the Symbian Signed team at Symbian who will be able to help you.

Does this place any additional restrictions on my application?

Once you have permissions from one or more manufacturers to use these capabilities on their devices, your application must be restricted by manufacturer so that it only installs onto the devices of those manufacturers from whom you have had approval.

© 2009 Symbian Foundation. This work is licensed under the Creative Commons Attribution No Derivatives 3.0 UK: England & Wales License.
Retrieved from "http://developer.symbian.org/wiki/index.php/Symbian_Signed_Test_Criteria_V4_Wiki_version"

Comments

Danm said…

This document is the master version of the Symbian Signed Test Criteria. As such, we don't encourage discussion of the content of the tests here at the end as that may cause confusion to developers reading these criteria for the first time.

Any comments left pointing out errors which need correcting in the criteria will be read and any appropriate changes made to the criteria.

Any other comments will be deleted, and discussions should take place in the Symbian Signed Forum (http://developer.symbian.org/forum/forumdisplay.php?f=17)

--Danm 12:59, 25 February 2010 (UTC)

Jlaidler said…

Test 2 should not refer to 'Task Manager'. The phrase 'Task List' should be used instead. Reasons - the Sogeti test results describe the test as 'The application can be closed by the Task List' and the Nokia Forum page describing what needs to be done in order to pass the test is titled 'Closing the application via Task List'.

--Jlaidler 15:24, 31 August 2010 (BST)

Sign in to comment…