A vendor audit should consist of a comprehensive review of a software vendor’s system development practices to ensure that the system was developed, implemented, tested and maintained in a quality manner.

Vendor Audits and the 483’s

Skipping the vendor audit used to be prevalent in the industry, as a result of numerous 483’s were being issued to various organizations the industry eventually caught onto the fact that the vendor audit is an essential aspect of dealing with any vendor large or small, external or internal.

What happens in reality?

In reality what vendor audits often end up being, are nice field trips off-site for a few days sometimes in a foreign country. Certainly there are rarely any benefits gained by either side from the audit – vendors or clients. This is a shame because the opportunity is certainly presented for bridging communications between both parties.

Good relationships pay off

I have seen companies that communicated well with their vendors and as a result with the next release of their system containing many of the enhancements that they would have paid for as customizations had the relationship not been as good. And vendors who receive good audit reports from vendors invariably get new clients who’ve heard good things about them. This is after all a very small industry and the same names and faces circulate over and over.

Good vendor bad vendor

Like all businesses there are good vendors and bad vendors, its up to you to find the difference between them as once you get locked into an agreement with a vendor its very difficult to break it as the application may already be implemented.

Do your research; ask other people in the industry about vendors they have used and their experience to date with them. Don’t just look at what the vendors give you – ask for documentation that is not on display.

Insist on a tour of their facilities, walk up to developers ask them what they are doing, don’t be afraid to this, if there was an FDA audit in your facility you can be sure that the FDA auditors won’t be shy about asking questions.

Ask them what they are doing if they are making a change ask to see the authorization for that change. This would be a design specification change if the system was not released or it would be a change control if the system was already released. If this is new code, ask to see what they are using as a specification. Is there any specification at all or if there is one is it too high level (Does it give too much scope to the developer to make a number of changes). Good technical specifications are not easy for a non-programmer to understand.

If you haven’t conducted a vendor audit before here’s a few tips to help.

System Development Life-Cycle Methodology

Programmers should not be just allowed to sit down and start typing in code – there should be a formal, approved, structured approach in place, as well as documented evidence that it has been followed. Minimally there should be requirements and functional specifications to describe what the software is supposed to do, and design specs stating how the software is supposed to do it.

There should be sufficient technical specifications and documentation to allow the ongoing support and maintenance of the system to continue, even in the absence of the system guru.

Design/Coding Standards

The vendor should have formal approved Programming Standards in place, and be able to provide evidence of adherence to same by the developers. There are specific requirements of programmers delineated by the FDA and just because they may be external vendors does not let you off the hook. Have one of the internal IT guys go on the audit and spend the entire day working with the vendor programmers.

Your programmer needs to check the quality of the design code (modular design no dead code) testing, peer reviews, and will get much better answers from another programmer than from the vendors slick sales talk.

The vendor should be able to demonstrate that the software has been tested thoroughly.
The answer you are looking for is that the tests are based on the specs, and all critical functions are in the specs. Now open a spec find a critical function and ask to see the testing for it.
Note that for documenting Unit/Integration/System/Acceptance/Regression Testing four things are required:

• What was tested
• What was the result
• Who did the testing
• When was the testing performed

Also verify that an adequate problem tracking mechanism is in place so that identified problems don’t fall through the cracks.
Have a programmer scroll through the module names while you watch. While the programmer is scrolling through point out a module and ask to see the code for that module. If they try to direct you to a different module, or try to give you a hardcopy of the code don’t allow them, they might have pretty version ready for the auditors.

Configuration Management

The vendor must have a method for controlling versions of the source code and for tracking what changes were included in what release. They must have a formal documented release process to ensure the right versions of the right modules were included in a new release. In addition, you must ensure that there is adequate configuration control in place to enable re-creation of earlier releases, if necessary. Ask to see an old release executable.

This capability is necessary in the event that you need to rollback to an earlier version of the system, if say the new version failed validation. It is also important that the vendor be able to bring up old versions of the system in order to support clients who because of the time involved in validation have chosen to stay with an earlier version of the system that they have successfully validated.

Change Control

The vendor must be able to prove that they have control over all changes made to the software. Verify that there is a written, approved, Change Control Procedure in place to prevent unauthorized changes and to ensure only tested modules are included in a release. Don’t just look at the samples that the vendors show you throughout the audit. Ask to be brought to their filing system for approved change control forms.

Flip through, find one and check it for compliance to industry and regulatory standards. This is what the FDA will do if your system results in a patient’s adverse reaction. Change control does not just apply to the software changes inherent in the system but also to the technical documentation surrounding the software such as user manuals.

Problem Reporting/Tracking

The hope is that the vendor has already an established method for reporting problems. Bring along some problems with you that you may have reported to the vendor and ask to see where these problems were reported and where and when the initial review process of these problems were been dealt with.
Follow the reporting process through to where a change control form was completed for each of the problems. Ask to see the programmer responsible for the fix and then get programmer to show you where the code was changed.

Examine the code

• Are there comments in the code explaining the change?
• Is the change control number included in the comment?
• Supportable code is well commented so that later programmers can make an educated decision regarding new modifications/fixes.

Written Procedures

It goes without saying that the vendor should have up-to-date approved, written procedures to control the development, testing, implementation, and maintenance of the software. There should be written approved procedures on creating SOPs that require periodic review of SOPs to ensure they stay up-to-date. There should be an SOP detailing the roles and responsibilities of the Quality Assurance Unit (QAU), and QAU SOPs for conducting internal compliance audits.

Security

The vendor must demonstrate that access to the software, the hardware and the development facility is controlled. Have someone show you the security method in place to ensure that code is protected from non-programmers, or unauthorized programmers. Ask an authorized programmer to sign-on and demonstrate that’s the code is accessible to them.

Quality Control

Verify that a unique quality assurance group is in place that conducts internal audits for compliance to standards. Find evidence of administrative action taken for non-compliance to internal or external standards. It isn’t enough for the vendor to have in-house compliance audits if the management isn’t going to do anything about the problems when they are found.

Backup and Contingency plans

The vendor must backup the software periodically, and have written procedures for making backups as well as restoring from backups. There should be a disaster recovery plan in place covering partial through total system loss, to ensure restoration of acceptable operation after a disruption. A Software Escrow Account should be in place to ensure your company gets a copy of the software and supporting documentation in the event of the vendor’s dissolution.

Qualification and Training

Each person involved in the development, testing and maintenance of the system must have the education, training, or experience to enable that person to perform their assigned functions. The evidence of qualification should be documented and available for review.
There should be a documented training program. Ask if someone fails their training.

Re-audit if necessary

In the old days it was assumed that if you went to outside vendors to obtain a system then you were relieved of the responsibilities of ensuring the software development was a quality process. Now we know better. Now it is time to dig deeper and harder when you go on a vendor audit, to turn up the real cracks in a process.

Once you have identified the vendor’s weakness, re-audit them in six months to check for improvement. If nothing has been done find another vendor. If you are too deep with this vendor, grab the business cards you gathered at the last Users Group Meeting for this system, and make some calls.

One client alone may not be enough to push a vendor into compliance, but when a substantial portion of their client base is screaming for compliance vendors tend to listen.