Skip to main content

What to Test?

When you are inside the code for an application, often this question doesn't get pondered.

 It seems like an obvious question, "What to test?" "Duh, everything!" or "What I just implemented." 

 But really, testing everything is often out of the question for a reasonably complex application, The permutations of states and settings grows quickly. If you have 4 settings which can each be one of two values, that's 16 (2 x 2 x 2 x 2) different combinations to be tested. Add another setting of only 2 choices and you have 32 combinations. If each setting is one of three values, there are 81 (3 x 3 x 3 x 3) combinations.  I think you can see the problem. If you have a good unit testing framework that will let you automate generation of numerous test cases, that can help a lot. But you still need to consider the question and the answer and decide if you are testing the right parts of the application , whether it's unit tests, manual tests, system tests, whatever.

 There are a number of different approaches you can think through to analyze your testing, which can help answer the question of what to test and put together a plan for effective testing. The first idea is to think about where risks are. Often this is in the newest code, the most complicated or the areas with the most demanding performance requirements.  Look for long routines and count the conditional statements if you want to make it a quantitative analysis. 

 Looking at the interfaces is a different angle for analysis. Not just application programming interfaces, but also network communications, shared memory, library calls, database queries and data acquisition are all types of interfaces. Interfaces are often the areas where there are more edge cases to test and where there is more chance for input values to wrong. 

 Often using probabilities can help focus testing efforts. What are the most likely code paths? Here you can make use of logging and historical information from your software.  Looking at your application from the outside-in or top-down plays well in terms of looking at features for testing.

There's always something to test. Of course, these analyses can also help inform your decisions on what not to test as well. This is equally important, as you want to avoid diminis hinges returns on your testing efforts. 

Good ideas on what to do instead of test is here: https://users.ece.cmu.edu/~koopman/des_s99/sw_testing/#alternative

Labels:

Comments

Post a Comment

Popular posts from this blog

You don't really know who you're talking to online...

The following is a story that I think highlights the assumptions that get you into trouble online... https://www.proofpoint.com/us/blog/threat-insight/i-knew-you-were-trouble-ta456-targets-defense-contractor-alluring-social-media This is particularly scary since we found so much utility in online connections during the pandemic and out of necessity, started trusting more online. Please note the timeline for this breach - it was a long, slow process, a key factor in many 'cons'. "Build trust" is a key first step, once someone has identified you as a party. You think...you're convinced you know who your talking to, but if you don't triangulate the identity with some non-online, ideally in-person information, you shouldn't trust. And even if you do get what seems like real-life confirmations of identity, you must look at questioning motives, needs, and keeping danger at arms-length. Online includes email, texting (sms), application chatbots, voice communicati...
Post a Comment
Read more

Threat Modeling Manifesto

Secure Your Code with Threat Modeling As a software developer, security should be a top priority. By proactively identifying and addressing potential vulnerabilities, you can significantly reduce the risk of breaches and data loss. What is Threat Modeling?   Threat modeling is a systematic approach to identifying, assessing, and mitigating security threats. It involves looking at your system from a hacker's perspective to uncover weaknesses and devise strategies to protect against attacks. See the  OWASP Cheat Sheet   Why is Threat Modeling Important? Proactive Security: By anticipating potential threats, you can take steps to prevent them. Risk Mitigation: Identify and address vulnerabilities before they can be exploited. Regulatory Compliance: Adhere to industry standards and regulations. Enhanced Security Posture: Strengthen your overall security posture. How to Get Started with Threat Modeling   The Threat Modeling Manifesto provides a valuable framewor...
Post a Comment
Read more

Where threat modeling can shine - an example from the EU MDCG-2019

From the  EU  MDCG 2019-16 Guidance on Cybersecurity for medical devices, December 2019 , this is the guidance on foreseeable risks.  Medical device manufacturers should ensure that a medical device is designed and manufactured in a way that ensures that the risks associated with reasonably foreseeable environmental conditions are removed or minimised. This may include the infield monitoring of the software’s vulnerabilities and the possibility to perform a device update (outside the context of a field safety corrective action) through, for example delivering patches to ensure the continued security of the device. During the risk management process, the manufacturer should foresee or evaluate the potential exploitation of those vulnerabilities that may be a result of reasonably foreseeable misuse. This, however, may depend on the specific situation. For example, using an unsecured memory-stick to enter data into a medical IT system can be considered “reasonably foreseeabl...
Post a Comment
Read more