S 2.82 Developing a test plan for standard software
Initiation responsibility: Top Management
Implementation responsibility: Head of IT, Head of Specialised Department
The test procedures described below are based on DIN ISO/IEC 12119 "Software Products, Quality Requirements and Testing Conditions", the Procedural Model for the Planning and Implementation of IT (V Model) and the Information Technology Security Evaluation Handbook (ITSEM), which are recommended as secondary literature.
Before deciding on a suitable software product, the products which were shortlisted according to the preselection (see S 2.81 Preselection of a suitable standard software product) must be obtained with a test licence and sufficiently tested. If it was not possible to test the product before purchase due to time limitations, internal procurement recommendations (adherence to internal standards) or any other reasons, tests must be conducted in any case before the product is finally put into operation. The results of these tests then form the basis for the installation regulations and other approval conditions.
Although the product requirements were checked in the preselection stage on the basis of the manufacturers' declarations, it cannot be assumed that these requirements are met to the desired extent. Rather, systematic testing must be conducted in order to check the suitability and reliability of the product on the basis of the requirements catalogue so that the most suitable product can then be selected.
It is recommended to divide these tests into four areas:
- Initial tests (tests for computer viruses, operation in the intended IT environment ....)
- Functional tests (test of the functional requirements)
- Tests of other functional features (compatibility, performance, interoperability, conformity with regulations or laws, user-friendliness, serviceability, documentation)
- Security-specific tests (check of the security requirements).
The basic procedure for testing standard software is shown in the following figure.
Figure 1: Basic procedure for testing standard software
Based on the shortlist created during the preselection stage, the products to be tested must be selected. A test plan is then developed.
This test plan includes the following contents:
- determining the contents of the test on the basis of the requirements catalogue
- checking the references
- determining the total testing time
- time planning, including time required for each test
- determining persons in charge of testing
- test environment
- contents of the test documentation
- determining decisive criteria.
These individual points are described in detail below.
Determining the contents of the test on the basis of the requirements catalogue
On the basis of the requirements catalogue, the requirements which are to be tested are selected. In particular, these should be the features which are of great importance or which have a high confidence requirement.
Checking the references
Initial references on the products to be tested were already obtained during the preselection stage (see S 2.81 Preselection of a standard software product). These can also be obtained if the external test group is trusted sufficiently.
If a certificate was issued for the product in accordance with the criteria for the evaluation of the security of IT systems (ITSEC) or the Common Criteria (CC), the certification report should be used to check to what extent the test results can be taken into consideration.
An internal test may not be required or could be conducted on a small scale. The free test capacities can be distributed to other tests.
Determining the total testing time
In order to limit the time required for testing, the total testing time should be determined in advance, e.g. in working days or by setting a deadline.
Time planning, including time required for each test
When testing several products, it is recommended to run comparative tests. This means that all products are tested by one test group regarding one requirement of the requirements catalogue. The testing time should thus be determined for each requirement of the requirements catalogue and is thus automatically distributed evenly to all products to be tested. The testing time results from the testing depth and complexity of the feature. The testing depth of the various features should be based on the confidence requirement, i.e. the amount of confidence that must be placed in the correct operation of this feature.
The proneness to error and frequency of use of the feature must also be taken into consideration. More detailed information is to be found in ISO 12119.
Notes:
- For security-related requirements, the test depth can also be adapted to the mechanism strength required.
- The testing time for the initial tests should be kept to a minimum in regard to the other tests.
The total testing time should then be distributed to the individual test sections in accordance with the relative testing time of the feature.
Determining persons in charge of testing
It should be determined for each test which tasks are to be carried out and who is responsible for these. In particular, it should be ensured that the Personnel/Supervisory Board, the Data Privacy Officer and the IT Security Officer are involved in some tests.
Test environment
Testing is always destructive as errors are being looked for. Tests should thus always be conducted in an isolated test environment.
If possible, the test environment should be a precise functional copy of the production environment. It is generally not viable to completely recreate the production environment.
To ensure that the same conditions are present for the selected products, a reference test environment should be defined. This can be further adjusted or limited for individual tests.
The resources required for the various tests (equipment, IT infrastructure) should be specified. It should be described in detail when, and to what extent, these must be available.
It is important that all operating systems in all versions (releases) used in production are available in the test environment. The intention is to determine system-based vulnerabilities of components of the production environment in connection with the standard software product to be installed. In exceptional cases, individual components can be omitted if aspects can be generalised.
The following aspects should be observed and help to set up a reliable and suitable test environment:
- An up-to-date virus scan program should ensure that the test environment is free of malware.
- The test environment must be free of side effects on the actual operation. In order to avoid interaction from the outset, the installation of dedicated IT systems is recommended.
- The access rights must be configured in the test environment in the same way as in the production environment.
- The access to the test environment must be controlled.
- When taken over into production, it must be ensured that the product is configured in the same way as in the test environment. A suitable integrity protection system (digital signatures, checksums) should thus be used in the test environment.
- The costs for setting up the test environment must be acceptable.
After all planned tests have been concluded, it should be decided whether the test environment is to be dismantled. It may be necessary for more tests to be carried out, i.e. it might be viable to retain the test environment. Before the test environment is dismantled, the test data should be deleted if they are no longer required (e.g. for installation at a later date). Printer products should be disposed of correctly, programs should be deinstalled. The test licences of the products which were not selected should be returned.
Contents of the test documentation
The test plan should state how detailed the test documentation should be. Here, aspects of comprehensibility, reproducibility and completeness should be taken into consideration.
The test documentation must contain test plans, targets, processes and results and it must also describe the correspondence between the tests and the specified requirements. All test activities and the test evaluation (including reasons for decisions) should be documented in writing. These include
- product name and description
- test begin, test end and testing time
- persons in charge of testing
- configuration of the test environment
- description of the test cases
- criteria for decisions, test results and argumentation
- unfulfilled requirements of the requirements catalogue.
The test group should be able to access clear documentation and records of the test activities and results (e.g. recording tool, forms etc.).
If an automatic tool is used for testing, the test documentation must contain sufficient information about this tool and its usage so that the decision can be understood.
Determining decisive criteria
When evaluating the contents of a test, the following three-point scale can be used, for example:
| Grade | Decisive criteria | |
|---|---|---|
| 0 | - | Requirements are not fulfilled. |
| or | ||
| - | Intolerable errors which cannot be eliminated are detected. | |
| 1 | - | Requirements are fulfilled, but there are still some reservations (e.g. function is only suitable to a limited extent). |
| or | ||
| - | Minor errors were detected. They are relatively unimportant, as they may only have a tolerable effect on production or as they may only occur with a negligible degree of probability. | |
| 2 | - | Requirements are fulfilled completely. |
| or | ||
| - | Errors which may have arisen must either be eliminated or have no effect on production. |
Table: Assessment scale
In the event that errors have arisen which cannot be reproduced, the tester must decide to which category (grade) the error should be allocated.
If errors have occurred which can be eliminated during testing, they should be tested again after elimination.
Example:
The example of a compression program in S 2.81 Preselection of a suitable standard software product is continued here in order to describe how the testing time can be determined for each requirement of the requirements catalogue. The testing time is based on the depth and complexity of the test. The confidence factor represents the amount of confidence in the feature.
The frequency of use, proneness to error and complexity of a feature are assessed as follows:
- 1 means "low"
- 2 means "moderate"
- 3 means "high".
An exception to this is if an unchangeable feature of the product is to be examined which is independent of the proneness to error and frequency of use. In this case, the value 0 is given. This results in the following table for the compression program:
Figure 2: Example of the compression program
In this example, the testing time is defined as follows:
Testing time = complexity * test depth
and
Test depth = confidence factor + proneness to error + frequency of use
(The percentage for the testing time in the last column of the table results from the values calculated for the testing time divided by the total of these values.)
An example for another method of calculating the testing time and assessing the test results is described in the ISO 12119 standard. Here, the individual requirements are weighted as follows: Assessment of each test = (complexity + proneness to error) * (frequency of use + importance).
In any case, the person responsible for testing must decide on an adequate assessment method for both the product and the organisation.
After developing a test plan, a tester or test group is appointed for each test specified in the test plan. The test group should be given the test plan and informed of the times specified for the individual tests.
Review questions:
- Is a test plan developed for testing standard software on the basis of the requirements catalogue?
- Is the test environment separated logically or physically from the production environment?
- Is there comprehensible, reproducible and complete test documentation?
- Does the test documentation contain test plans, targets, processes and results and describe the correspondence between the tests and the specified requirements?