The foundation of a solid validation effort is the requirements for your intended use. If you have weak, incomplete, and inconsistent requirements, you will struggle with the validation effort. Take the time to ensure your requirements are complete, concise, unambiguous, and verifiable.
This can take the form of a customer specification or a user specification but needs to be documented and managed. Even if you are validating a small spreadsheet macro, it’s important to understand what the intent of the application encompasses.
The Use of Shall
Use “shall” to specify requirements. Doing so makes verifiable requirements easy to find. Each requirement should contain only one shall. If you find yourself having more than one ‘shall’ in a requirement, consider breaking the requirement up into multiple requirements.
The following table provides some keywords that can alert you to potentially unverifiable requirements (not intended to be a complete list):
Also be on the lookout for “run-on” requirements. A specification developer might think a requirement such as the following is concise:
“The software shall provide a login page that contains the company logo and a field for password entry that displays stars (“*”) when a character is entered and then transitions to the order entry page upon accepted password or raises a “try again?” message if the password is incorrect.”
So where do you start to verify that?
How do you know if all aspects are verified? Breaking that up into more focused ‘chunks’ will help ensure testability and traceability but will also likely force thought on what is truly a requirement and what is design.
Don’t Create Requirements That Add No Value
It’s easy to fall into requirements traps that forces testing that really adds little value.
There are often cases where requirements are “statements of fact” that really aren’t requirements that need to be verified. An example might be “The user interface module shall contain all the user interface code.” Yes, you could verify this, but why? And what would you do if it failed? Completely restructuring the code would greatly increase risk so it’s likely nothing would be done!
So be on the lookout for “statements of fact” and consider scrubbing them from the (verifiable) requirements. They can still be stated in a specification, as design expectations or goals,
just not as verifiable statements. Similarly, if a requirement indicates the system is to be “fast,” this is clearly subjective.
How do you verify this?
You could verify that response time is within 10 seconds but if I really need it within 5 seconds, it might ‘meet’ requirement but doesn’t meet your needs and thus your test effort was wasted.
Baseline Your Requirements
Requirements should be baselined before embarking on a validation effort. To baseline requirements, a review needs to be held. The review is key to ensuring solid requirements.
Reviewers should include system architects (to confirm the requirements represent and can support the
architecture), developers (to ensure requirements can be developed), testers (to confirm the requirements can be verified), and quality assurance (to ensure that the requirement are complete). If requirements aren’t verifiable, your testing will be meaningless.
Without baselining requirements, you can be in a situation where the rug is constantly being pulled out from under you. This is extremely inefficient and can lead to chaos. Following a baseline, changes should be controlled and managed, allowing impact assessment and a controlled test adjustment.
Product Software
For product software, requirements often take the form of (more general) customer specifications and then greater specificity in a system specification. This allows the system to be ‘shaped’ from the customer or end-user view and then the system specification puts the meat on the bones.
Generally, validation testing efforts show that the system meets those customer / user needs and verification shows compliance to system specifications.
Organize Requirements
Most systems have many requirements. Some, like use cases, may be less obvious than the traditional “shall” type statements but all should be considered. If there are a lot of requirements (probably any number over 30), you’ll want to partition the requirements into logical groups to better
facilitate the test effort. This allows for shorter, more focused test protocols.
This also allows for test developers with specific expertise to be assigned protocol development more in line with their expertise.
Systems are typically set up as architectural subsystems; e.g., Graphical User Interface (GUI), database, interface to other systems, etc. These may be reasonable ways to partition the test effort. Make sure to do a sanity check.
For example, a database intensive system may have a small interface to an external application. It wouldn’t make sense to just separate testing by external interface (maybe 10% of the system) and database (maybe 90% of the system).
It may make sense in this case to have the external application interface as one group but then further decompose the database portion into multiple, smaller groupings.
