10 Commandments of Software Testing

"Software Testing is a systematic activity but it also involves economics and human psychology." - Glenford J. Myers

Economics of software testing is to determine and predict the faults of the system early by using foreseeable models and applying structured test strategies and test methodologies to discover those at early phases of the software development life cycle.

Psychology of testing is to destructively test the application by identifying as many exceptional or out of the box scenarios or sometimes called as the third vision.

A set of good test scenarios evaluates every possible permutations and combinations of a program during ideal conditions. In addition, Software Test Engineer needs the proper vision to successfully test a piece/whole application to comply with the Standards and the Quality.

Whenever Software Test Engineer tests a program, they should add some value in it rather than performing only the requirements conformance and validation. A systematic and well planned test process adds the value of quality and reliability of the software program.

The most important considerations in the software testing are the issues of psychology, leading to the set of principles and guidelines to evaluate software quality:

1. Testing is the process of experimenting a software component by using a selected set of test cases, with the intent of revealing defects and evaluate quality Software Test Engineer executes the software using test cases to evaluate properties such as reliability, usability, maintainability, and level of performance. Test results are used to compare the actual properties of the software to those specified in the requirements document as quality goals.

Deviations or failure to achieve quality goals must be addressed. Software Testing has a broader scope rather being only limited to the execution of program or detecting errors more rigorously as described in the test process improvement models such as TMMi framework models.

2. When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yet undetected defect(s).

Principle 2 supports a strong and robust designing of test cases. This means that each test case should have a goal to identify or detect a specific type of defect. Software Test Engineer approaches the scientific method of designing the tests to prove or disapprove the hypothesis by the means of test procedures.

3. A test case must contain the expected output or result. Principle 3 supports the fact that a test case without expected result is of zero value. Expected output or result determines whether a defect has been revealed or the conditions have been passed during the execution cycle.

4. Test cases should be developed for both valid and invalid input conditions. Use of test cases that are based on invalid inputs is very useful for revealing defects since they may exercise the code in unexpected ways and identify unexpected software behaviour. Invalid inputs also help developers and Software Test Engineers to evaluate the robustness of the software, that is, its ability to recover when unexpected events occur (in this case an erroneous input).

For example, software users may have misunderstandings, or lack information about the nature of the inputs. They often make typographical errors even when complete/correct information is available. Devices or software program may also provide invalid inputs due to erroneous conditions and malfunctions.

5. The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component. Principle 5 supports to the fact that the higher the number of defects already detected in a component, the more likely it is to have additional defects when it undergoes further testing.

For example, if there are two components A and B, and Software Test Engineers have found 20 defects in A and 3 defects in B, then the probability of the existence of additional defects in A is higher than B. This empirical observation may be due to several causes and degree of influence of the software factors. Defects often occur in clusters and often in code that has a high degree of complexity and is poorly designed.

6. Test Cases must be repeatable and reusable. Principle 6 is utmost important and plays a vital role supporting the fact that it is also useful for tests that need to be repeated after defect repair. The repetition and reuse of tests is also necessary during regression test (the retesting of software that has been modified) in the case of a new release of the software.

7. Testing should be carried out by a group that is independent of the development group. This principle holds true for psychological as well as practical reasons and does not say that it is impossible for a programming organization to find some of its errors, because organizations do accomplish this with some degree of success. Rather, it implies that it is more economical for testing to be performed by an objective, independent party which gives a direction of the third vision by the means of test cases. Finally, independence of the testing group does not call for an adversarial relationship between developers and Software Test Engineers.

8. Test Activities should be well planned. Test plans should be developed for each level of testing, and objectives for each level should be described in the associated plan. The objectives should be stated as quantitatively as possible. Plans, with their precisely specified objectives, are necessary to ensure that adequate time and resources are allocated for testing tasks, and that testing can be monitored and managed.

Test planning must be coordinated with project planning. A test plan is a roadmap for the testing which should be mapped to organizational goals and policies pertaining to the software program.

Test risks must be evaluated at each levels of testing. Careful test planning avoids wasteful "throwaway" tests and unproductive and unplanned "test-patch-retest" cycles that often lead to poor-quality software and the inability to deliver software on time and within budget.

9. Avoid throwaway test cases unless the program is truly a throwaway program. Whenever the program has to be tested, the test cases must be reinvented. More often than not, since this reinvention requires a considerable amount of work, Software Test Engineer tends to avoid it. Therefore, the retest of the program is rarely as rigorous as the original test, meaning that if the modification causes a previously functional part of the program to fail, this error often goes undetected.

10. Test results should be inspected meticulously. This is probably the most obvious principle, but again it is something that is often overlooked.

Weve seen numerous tests that show many subjects failed to detect certain errors, even when symptoms of those errors were clearly observable on the output listings. Put another way, errors that are found on later tests are often missed in the results from earlier tests.

For example:
I. A failure may be overlooked, and the test may be granted a "PASS" status when in reality the software has failed the test. Testing may continue based on erroneous test results. The defect may be revealed at some later stage of testing, but in that case it may be more costly and difficult to locate and repair.

II. A failure may be suspected when in reality none exists. In this case the test may be granted a "FAIL" status. Much time and effort may be spent on trying to find the defect that does not exist. A careful re-examination of the test results could finally indicate that no failure has occurred.

Summarizing all the above principles, Testing is an extremely creative and intellectually challenging task. Creativity required in testing a large program exceeds the creativity required in designing or developing that program. We already have seen that it is impossible to test a program sufficiently to guarantee the absence of all errors. This requires a systematic test process and methodologies to design the robust test cases.

This principle supports that:

1. A Software Test Engineer needs to have comprehensive knowledge of the software engineering discipline.

2. A Software Test Engineer needs to have knowledge from both experience and education as to how software is specified, designed, and developed.

3. A Software Test Engineer needs to have knowledge of fault types and where faults of a certain type might occur in code constructs.

4. A Software Test Engineer needs to reason like a scientist and propose hypotheses that relate to presence of specific types of defects.

5. A Software Test Engineer needs to have a good grasp of the problem domain of the software that he/she is testing.

6. A Software Test Engineer needs to create and document test cases. To design the test cases the Software Test Engineer must select inputs often from a very wide domain.

7. A Software Test Engineer needs to design and record test procedures for running the tests.

8. A Software Test Engineer needs to execute the tests and is responsible for recording results.

9. A Software Test Engineer needs to analyze test results and decide on success or failure for a test. This involves understanding and keeping track of an enormous amount of detailed information.

10. A Software Test Engineer needs to know the method for collecting and analyzing test related measurements.

References:
The principles and the excerpts have been taken from the following books:
1. The Art of Software Testing by Glenford J. Myers
2. Software Testing Principle by Bertrand Meyer
3. Practical Software Testing by Ilene Burnstein
4. TMMi Framework by TMMi Foundation.