Testing Process and Limitations of Testing

Software Testing process

Software testing process

1. Test Strategy and Test Plan
Every project needs a Test Strategy and a Test Plan. These artifacts describe the scope for testing for a project:

• The systems that need to be tested, and any specific configurations
• Features and functions that are the focus of the project
• Non-functional requirements
• Test approach—traditional, exploratory, automation, etc.—or a mix
• Key processes to follow – for defects resolution, defects triage
• Tools—for logging defects, for test case scripting, for traceability
• Documentation to refer, and to produce as output
• Test environment requirements and setup
• Risks, dependencies, and contingencies
• Test Schedule
• Approval workflows
• Entry/Exit criteria

And so on… Whatever methodology your project follows, you need to have a Test Strategy and Software Testing Plan in place. Make them two separate documents, or merge them into one.

Without a clear test strategy and a detailed test plan, even Agile projects will find it difficult to be productive. Why you ask? Well, the act of creating a strategy and plan brings out a number of dependencies that you may not think of otherwise.

For example, if you’re building a mobile app, a test strategy will help you articulate what Operating Systems (iOS/Android), OS versions (iOS 7 onwards, Android 4.4 onwards, etc.), devices (last three generations of each type of iOS device, specific models of Android devices) you need to test the app for.

Usually, a Functioning Organisation Will Have Nailed Their Device And OS Support Strategy, And Review It Quarterly to keep up with the market; test managers creating a strategy or plan for their project will help validate the enterprise-wide strategy against project-specific deliverables.
Testing performs this all-important gatekeeping function and Helps Bring Visibility to any issues that may be brushed under the carpet otherwise.

2. Test Design
Now that you have a strategy and a plan, the next step is to dive into creating a test suite. A test suite is a collection of test cases that are necessary to validate the system being built, against its original requirements.

Test design as a process is an amalgamation of the Test Manager’s experience of similar projects over the years, testers’ knowledge of the system/functionality being tested, and prevailing practices in testing at any given point. For instance, if you work for a company in the early stages of new product development, your focus will be on uncovering major bugs with the alpha/beta versions of your software, and less on making the software completely bug-proof.

The product may not yet have hit the critical “star” or “cash cow” stages of its existence—it’s still a question mark. And you probably have investors backing you, or another product of your own that is subsidizing this new initiative until it can break even. Here, you’re trying to make significant strides—more like giant leaps—with your product before you’re happy to unwrap it in front of customers. Therefore, you’re less worried about superficial aspects like look and feel, and more worried about fundamental functionality that sets your product apart from your competitors.

In such a scenario, you may use lesser negative testing and more exploratory or disruptive testing to weed out complex, critical bugs. And you may want to leave out the more rigorous testing until you have a viable product in hand. So your test suite at the beginning of the product lifecycle will be tuned towards testing fundamentals until you’re close to releasing.

When you are happy to release a version to your customers, you’ll want to employ more scientific testing to make it as bug-free as possible to improve customer experience. On the other hand, if you’re testing an established product or system, then you probably already have a stable test suite. You then review the core test suite against individual project requirements to identify any gaps that need additional test cases.

With good case management practices, you can build a test bank of the highest quality that helps your team Significantly Reduce Planning And Design Efforts.

3. Test Execution
You can execute tests in many different ways—as single, waterfall SIT (System Integration Test) and UAT (User Acceptance Test) phases; as part of Agile sprints; supplemented with exploratory tests; or with test-driven development. Ultimately, you need to do an adequate amount of software testing to ensure your system is (relatively) bug-free.

Let’s set methodology aside for a second, and focus on how you can clock adequate testing. Let’s go back to the example of building a mobile app that can be supported across operating systems, OS versions, devices. The most important question that will guide your test efforts is “what is my test environment?”.

You need to understand your test environment requirements clearly to be able to decide your testing strategy. For instance, does your app depend on integration with a core system back end to display information and notifications to customers? If yes, your test environment needs to provide back-end integration to support meaningful functional tests.

In this example, we’re talking about doing front-end tests with a dummy back end to support in-sprint testing, and wait until an integrated test environment is ready. It is common practice to schedule integration tests just after delivery sprints and before release. Your team can then run a dedicated System Integration Test, focusing on how the app components work with the back end to deliver the required functionality. So while app-specific bugs will primarily be reported during the sprints, functional end-to-end bugs will crop up during the integration test. You can follow this up with a UAT cycle to put finishing touches in terms of look and feel, copy, etc. How your team executes test cycles depends on the enabling infrastructure, project, and team structure in your organization.

4. Test Closure
Right—so you have done the planning necessary, executed tests and now want to green-light your product for release. You need to consider the exit criteria for signaling completion of the test cycle and readiness for a release. 

Let’s look at the components of exit criteria in general:

• 100% requirements coverage: all business and technical requirements have to be covered by testing.
• Minimum % pass rate: targeting 90% of all test cases to be passed is the best practice.
• All critical defects to be fixed: self-explanatory. They are critical for a reason.

As a rule of thumb, I’ve seen projects mandate a 90% pass rate and all critical defects being fixed before the team can move on to the next phase of the project. And on big transformation initiatives, I’ve seen individual releases move to the next phase (to aid beta pilots) with as little as 80%, with the understanding that the product won’t reach the customer until mandatory exit criteria are met. Ultimately, what works for your team is down to your circumstances and business demands.

Limitations of software testing

Software testing possesses some limitations those are:

• One cannot test a program completely.
• One can only test against system requirements.
• May not detect errors in the requirements. 
• Incomplete or ambiguous requirements may lead to inadequate or incorrect testing.
• Exhaustive (total) testing is impossible in the present scenario.
• Time and budget constraints normally require very careful planning of the testing effort.
• Compromise between thoroughness and budget.
• Test results are used to make business decisions for release dates.
• Even if you do find the last bug, you’ll never know it.
• One will run out of time before you run out of test cases.
• One cannot test every path.
• One cannot test every valid input.
• One cannot test every invalid input.

No absolute proof of correctness: Unless a formal specification can be shown to be correct and, indeed, reflects exactly the user's expectations, no claims of product correctness can be made.

Next: Overview of Graph Theory

Credit: Reqtest