This one is tricky and by its nature highly controversial, deviations are not bad things when handled correctly they are a legitimate part of any system testing and should not be looked upon as bad work by the analyst.
When an auditor assesses a system, they will rely on years of experience validating the same types of systems, (even sometimes the same systems). It is the “sick gut feel” the auditor gets when looking at an application of over 2 million lines of code and only finds five documented Deviations. In a system of that size and complexity, unless the system was developed and debugged according to Military Standards, there aren’t enough Deviations to show a thorough Validation Testing effort.
Auditors
For instance a knowledgeable auditor will expect to see somewhere in the range of 50-100 failures documented for LIMS, or an auditor will assume that either the tests or the testers were inadequate. In a small system, 1-20 failures is the norm. If there are 100 failures in a small system, this indicates that the system hasn’t been debugged enough prior to Validation start. On the other hand, if there are only 10 failures on a system as complex as LIMS, this is a sure indication that either failures are not getting recorded by the testers, or the tests were not adequate to uncover the bugs in the system.
One of the most important goals of validation testing is to identify the bugs and determine their limits and a feasible workaround before the system goes into production. In this way, the safety and efficacy of the product is not compromised. In short, if you don’t find the bugs, you aren’t going to be able to devise a suitable workaround.
The Real World
Here’s an actual finding made during an External Audit:
“Additional testing is needed, Even though there is a large volume of test documentation for this system, test coverage is poor for critical functions (e.g Security). In addition, there is little or no field verification with invalid/erroneous/borderline data, and there is insufficient testing of error-handling capability. In general, there was insufficient testing to identify the problems with the system. This was evidenced by the small number of system problems identified during the validation effort. In a system as large and complex as System X, with as many problems as are listed in the vendor’s Known Problem List, a validation effort should be able to highlight a sizeable number of problems. If only a few problems were found during the testing, this is a sure indication that either failures were not getting documented by the testers, or the test suite was insufficient to uncover the bugs in the system.”
Qualified People
Assign qualified people to validate your system and give them support, time and resources to conduct comprehensive testing of your system. Then don’t minimise or rationalize the Deviations found during the validation effort. Deviations are evidence that the system was challenged thoroughly and completely, and properly documented Deviations speak well of your integrity and intent.
