Limits of Logic

Formal methods in software were the answer to the problem. The idea was you could prove that a program had no bugs. Which would have been a good thing.

However. The longest program was a couple of hundred lines or so. Whereas useful programs are many hundreds of thousands of lines long and really useful programs are millions of lines long.

Testing is also a good way of establishing the "truth" of a program.

(From Glenford J. Myers, The Art of Software Testing (Wiley, 1979))

"Self-Assessment"

Statement of Requirements:

The program reads three integer values from a card.

The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

Your Exercise Requirements:

On a sheet of paper, write a set of test cases (i.e., specific sets of data) that you feel would adequately test this program.

The next step is an evaluation of the effectiveness of your testing. It turns out that this program is more difficult to write than it first appears.

As a result, different versions of this program have been studied, and a list of common errors has been compiled. Evaluate your set of test cases by using it to answer the following questions. Give yourself one point for each "yes".

Answers on next page.

1. Do you have a test case that represents a valid scalene triangle? (Note that test cases such as 1,2,3 and 2,5,10 do not warrant a "yes" answer, because there does not exist a triangle having such sides.)

2. Do you have a test case that represents a valid equilateral triangle?

3. Do you have a test case that represents a valid isosceles triangle? (A test case specifying 2,2,4 would not be counted.)

4. Do you have at lest three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., 3,3,4; 3,4,3; and 4,3,3)?

5. Do you have a test case in which one side has a zero value?

6. Do you have a test case in which one side has a negative value?

7. Do you have a test case with three integers greater than zero such that the sum of two of the
numbers is equal to the third? (That is, if the program said 1,2,3 represents a scalene triangle, it would contain a bug.)

8. Do you have at lest three test cases in category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the other two sides (e.g., 1,2,3; 1,3,2; and 3,1,2)?

9. Do you have a test case with three integers greater than zero such that the sum of two of the
numbers is less than the third (e.g., 1,2,4 or 12,15,30)?

10. Do you have at least three test cases in category 9 such that you have tried all three permutations (e.g., 1,2,4; 1,4,2 and 4,1,2)?

11. Do you have a test case in which all sides are 0 (i.e., 0,0,0)?

12. Do you have at least one test case specifying non-integer values?

13. Do you have at least one test case specifying the wrong number of values (e.g. two, rather than three, integers)?

14. For each test case, did you specify the expected output from the program in addition to the input?

Of course, a set of test cases that satisfies the above conditions does not guarantee that all possible errors would be found, but since questions 1-13 represent errors that have actually occurred in different versions of this program, an adequate test of this program should expose these errors.

If you are typical, you have done poorly on this test. As a point of reference, highly experienced
professional programmers score, on the average, only 7.8 out of a possible 14.

The point of the exercise is to illustrate that the testing of even a trivial program such as this is not an easy task. And if this is true, consider the difficulty of testing a 100,000-statement air-traffic-control system, a compiler, or even a mundane payroll program.

Taken from: http://www.lsc-group.phys.uwm.edu/daswg/docs/informal/LIGO_Software_QA.pdf

Now my question is this. If we cannot use logic to demonstrate the "truth" of something we have total control over how can we possible use logic to demonstrate the "truth" about our observations of the real-world?