Skip to main content

What to Test?

When you are inside the code for an application, often this question doesn't get pondered.

 It seems like an obvious question, "What to test?" "Duh, everything!" or "What I just implemented." 

 But really, testing everything is often out of the question for a reasonably complex application, The permutations of states and settings grows quickly. If you have 4 settings which can each be one of two values, that's 16 (2 x 2 x 2 x 2) different combinations to be tested. Add another setting of only 2 choices and you have 32 combinations. If each setting is one of three values, there are 81 (3 x 3 x 3 x 3) combinations.  I think you can see the problem. If you have a good unit testing framework that will let you automate generation of numerous test cases, that can help a lot. But you still need to consider the question and the answer and decide if you are testing the right parts of the application , whether it's unit tests, manual tests, system tests, whatever.

 There are a number of different approaches you can think through to analyze your testing, which can help answer the question of what to test and put together a plan for effective testing. The first idea is to think about where risks are. Often this is in the newest code, the most complicated or the areas with the most demanding performance requirements.  Look for long routines and count the conditional statements if you want to make it a quantitative analysis. 

 Looking at the interfaces is a different angle for analysis. Not just application programming interfaces, but also network communications, shared memory, library calls, database queries and data acquisition are all types of interfaces. Interfaces are often the areas where there are more edge cases to test and where there is more chance for input values to wrong. 

 Often using probabilities can help focus testing efforts. What are the most likely code paths? Here you can make use of logging and historical information from your software.  Looking at your application from the outside-in or top-down plays well in terms of looking at features for testing.

There's always something to test. Of course, these analyses can also help inform your decisions on what not to test as well. This is equally important, as you want to avoid diminis hinges returns on your testing efforts. 

Good ideas on what to do instead of test is here: https://users.ece.cmu.edu/~koopman/des_s99/sw_testing/#alternative

Labels:

Comments

Post a Comment

Popular posts from this blog

Unit Testing - What to Test

This I wrote to answer a question that came up when we were discussing our software process and I was training developers on how to unit test. It seems a simple enough question, but I kept pondering it and delving deeper until I realized I needed to write this monograph. What unit tests should we write? How do we know what to test? Ideally, unit tests should cover every path through the code. It should be your chance to see every path through your code works as expected and as needed. If you are practicing Test Driven Development then it's implied everything gets a test. In the real world, you might not be allowed to test everything - for instance, if the testing suite ends up taking a week to run, then the world will have changed by the time it finishes and the test results will be obsolete. Unit testing at it's basic is testing an object, a method - the smallest unit of your code that it can test independently. It should test the inputs "goes into" an
Post a Comment
Read more

Healthcare and Health Informatics Glossary

Here is a glossary of terms useful in Healthcare and Health Informatics ACO (Accountable Care Organization) MEDICARE’s outcomes-based contracting approach Arden Syntax an approach to specifying medical knowledge and clinical decision support rules in a form that is independent of any EHR and thus sharable across hospitals ARRA (American Recovery and Reconstruction Act) the Obama administration’s 2009 economic stimulus bill Blue Button an ASCII text based standard for heath information sharing first introduced by the Veteran’s Administration to facilitate access to records stored in VistA by their patients. The newer Blue Button + format provides both human and machine readable formats. CCD (Continuity of Care Document) an XML-based patient summary based on the CDA architecture CCOW (Clinical Context Object Workshop) an HL7 standard for synchronizing and coordinating applications to automatically follow the patient, user (and other) contexts to allow the clinical u
2 comments
Read more

Files as UI

Files as UI vs API  -  compares attributes of iCloud vs Dropbox. It starts on an interesting note - the model of a file system in the UI is dying, and should be let go. Beyond that it looks at mappings of each system to a file system from an API point of view and compares the successes of each. I find the initial thread the most interesting. Drop the mental model of a file system - which maps virtual concepts of files and directories to a physical model of papers, folders and file cabinets - and replace it with...what? This is a paradigm shift for me. I have to admit, I loath, hate, nay, despise looking for things. If I can't find something easily, it's only about a minute before I start growling and muttering things my mother would disapprove of. On this basis, I like the idea that I can save myself from thinking about where to put things or, where I have already put them. But how do we do this? It's non-trivial, since humans think of "things" and once they
2 comments
Read more