Quality Vs Validation – Protocol Acceptance Criteria

  • 0

So the Validation Department generate the protocol and the Quality Department are involved in reviewing the protocol. Having worked on various different projects it has become quite apparent that the quality review on any protocol is a very important review as it irons out all of the documentation mistakes that occur when the document is being generated.

Quality Vs Validation  - Good Vs Evil
Quality Vs Validation - Good Vs Evil

On the other hand I often wonder if quality takes this review too far, and if they do why?

Quality Review

For example if you have a quality person reviewing an IQ or OQ protocol on a computer application, is it acceptable for them to review it having little knowledge of the application.

Does this lead to the quality person reviewing the protocol with an overly cautious approach to the point where the validation engineer is ready to pull their hair out in frustration.

A Simple Scenario

Lets take a very simple scenario. A test protocol usually has the following fields:

  • Test Procedure
  • Acceptance criteria
  • Meets Acceptance Criteria Yes/No
  • Performed by /Date

The column to take note of in this scenario is the Acceptance Criteria and the Performed by/Date field.

Lets discuss the acceptance criteria field first, this field usually contains the criteria that must be met in order to pass a test section. If the test is carried out and the acceptance criteria are not met then this usually results in a deviation or an event being raised.

I don’t think anyone has an issue with this scenario, but in terms of acceptance criteria what is acceptable?

Let take a simple example:

Test Procedure: Click the Yes radio button
Acceptance Criteria: Deviation message is displayed
Meets acceptance criteria: Yes
Performed by: Joe Soap 12/12/09

Is this acceptable from a quality perspective though, I mean when you sign the performed by section do you mean that the test has been completed or do you mean that the deviation message was displayed and the test was completed.

Screenshots

If the quality department are not happy with the latter then they will require a screenshot as evidence that the deviation message was displayed.

Do you see what I am getting at here; does this mean that screenshots are required for all acceptance criteria?

If so then your protocol will contain numerous attachments and a long review time.

Before your protocol is approved you really need to work closely with the quality department in order to understand what is acceptable from an acceptance criteria view point.

If you would like further assistance with protocol generation please feel free to contact Premier Validation

0
shares

  • maryacton

    Nice post – Yes we have the same issues with quality, they are important to the overall process but it is important that validation and quality work together and not think they are on different teams.

  • maryacton

    Nice post – Yes we have the same issues with quality, they are important to the overall process but it is important that validation and quality work together and not think they are on different teams.

  • S. J. D’Souza

    Documentation corrections do not equal spelling mistakes. There is more to a quality review that just spell check. On the surface it seems so much easier to have to review a document than having to create it in the first place. Having to write a protocol for anyone to understand decreases the efficiencies of the risk based approach or quality by design. IMHO it is incumbent of the quality approver to educate him/herself of the application in question before/while approving. Approval should be an iterative process to deliver the best qualification document.

  • S. J. D’Souza

    Documentation corrections do not equal spelling mistakes. There is more to a quality review that just spell check. On the surface it seems so much easier to have to review a document than having to create it in the first place. Having to write a protocol for anyone to understand decreases the efficiencies of the risk based approach or quality by design. IMHO it is incumbent of the quality approver to educate him/herself of the application in question before/while approving. Approval should be an iterative process to deliver the best qualification document.

  • gokeeffe

    Hi S. J. D’Souza,

    Thanks for the comments, in your experience could you provide a list of other aspects of a quality review?

    Cheers

    Graham

  • gokeeffe

    Hi S. J. D’Souza,

    Thanks for the comments, in your experience could you provide a list of other aspects of a quality review?

    Cheers

    Graham

  • snaranjo

    Quality involvement on Computer validation is essential to success, but also to concentrate on relevant testing and documentation. We can take tons of screenshots as evidence of standard functionalities (hardly tested by the supplier) and miss the opportunity to test critical or risky configuration. Then we have a very nice but useless documentation. In other words, validation effort should be focussed on demonstrating that system (or system configurations) meet the user processes without compromising electronic records. Documentation is a part, not the goal of validation, and quality people not always has this view. In my opinion, testing must be done by a well-trained or experienced user who write “pass/not pass” and sign the execution. A second user (witness or not) review the results and take part on resolution of fails/non conformities. We use to involve QA as final test reviewer (and protocol/report approvers).

    Regards

  • snaranjo

    Quality involvement on Computer validation is essential to success, but also to concentrate on relevant testing and documentation. We can take tons of screenshots as evidence of standard functionalities (hardly tested by the supplier) and miss the opportunity to test critical or risky configuration. Then we have a very nice but useless documentation. In other words, validation effort should be focussed on demonstrating that system (or system configurations) meet the user processes without compromising electronic records. Documentation is a part, not the goal of validation, and quality people not always has this view. In my opinion, testing must be done by a well-trained or experienced user who write “pass/not pass” and sign the execution. A second user (witness or not) review the results and take part on resolution of fails/non conformities. We use to involve QA as final test reviewer (and protocol/report approvers).

    Regards

  • Several interesting points should be clarified:

    1) prior to writing a protocol, I recommend that a team be established including the writer, the system owner, and the quality reviewer. This allows the team members to learn their strengths, weaknesses, and personality traits (if the quality reviewer is weak in computer validation the others can diplomatically provide suggestions, learn that the quality reviewer is not open to suggestions, or learn what the quality reviewer expects). Having written over a hundred protocols, I highly recommend this.

    2) prior to writing a protocol, I recommend that the writer generate an outline of what will be tested and identify the template to be used (to achieve team consensus). For example, I recommend that you include columns for “Expected Results” and “Actual Results” so that the reviewer can determine if actual results met expected results.

    3) getting to know the team members before writing the protocol improves the likelihood of it being right-first-time (minimizing protocol approval time and protocol review time after execution).

    4) it is important that a protocol be written such that execution and review is not open to interpretation (the team, including the executor, knows what is expected as does the reviewer). As an example, state when a screenshot is required (attach printout demonstrating that ….). Screenshots are required to demonstrate tasks have been performed successfully. As a rule of thumb, one or two, per test case.

    If you have any questions, you can reach me via email at: jeffreygassman@validationplusinc.com

  • Several interesting points should be clarified:

    1) prior to writing a protocol, I recommend that a team be established including the writer, the system owner, and the quality reviewer. This allows the team members to learn their strengths, weaknesses, and personality traits (if the quality reviewer is weak in computer validation the others can diplomatically provide suggestions, learn that the quality reviewer is not open to suggestions, or learn what the quality reviewer expects). Having written over a hundred protocols, I highly recommend this.

    2) prior to writing a protocol, I recommend that the writer generate an outline of what will be tested and identify the template to be used (to achieve team consensus). For example, I recommend that you include columns for “Expected Results” and “Actual Results” so that the reviewer can determine if actual results met expected results.

    3) getting to know the team members before writing the protocol improves the likelihood of it being right-first-time (minimizing protocol approval time and protocol review time after execution).

    4) it is important that a protocol be written such that execution and review is not open to interpretation (the team, including the executor, knows what is expected as does the reviewer). As an example, state when a screenshot is required (attach printout demonstrating that ….). Screenshots are required to demonstrate tasks have been performed successfully. As a rule of thumb, one or two, per test case.

    If you have any questions, you can reach me via email at: jeffreygassman@validationplusinc.com

  • Excellent comments – particularly the detail from Jeff.

    Something to keep in mind is that there are several common rationales or approaches to validation. These can typically vary by company, by division within a company, or even by the project that you are representing. Understanding the validation approach for your particular project is critical if you expect to be successful when it comes time for a quality review. Deliverable types that were acceptable on your last project may no longer apply, even for the same type of system.

    For example, if your project follows a risk-based approach there will be specific criteria that quality uses as a measuring stick. You may have a full suite of system lifecycle documentation, such as User Requirements, Functional Specifications, and/or a Design Specification of some sort. Depending on how the risk-based approach is structured, quality may expect your protocol to include verifications of specific elements from within these lifecycle documents. Your role in a project may be focused on protocol development, however, it’s critical that you understand validation rationales that may already be planned for your project. This is in alignment with comments by Jeff that you should familiarize yourself with quality’s expectations.

    To summarize, while it’s crucial that protocols and test cases be structured in a manner that eliminates ambiguity, understanding the rationales for testing and the appropriate deliverables that feed into the qualification effort are foundational when it comes time for a quality review.

    As a caveat, there are clearly organizations who don’t require indepth planning or advanced rationales when planning a validation effort. Whereas understanding a risk-based approach to protocol writing is critical if you want to be successful in a risk-based environment, communication of expectations is equally critical in a less rigorous business environment. In either case, the key to successful protocol development is understanding the basis that quality will be using when it’s time to review your work.

  • Excellent comments – particularly the detail from Jeff.

    Something to keep in mind is that there are several common rationales or approaches to validation. These can typically vary by company, by division within a company, or even by the project that you are representing. Understanding the validation approach for your particular project is critical if you expect to be successful when it comes time for a quality review. Deliverable types that were acceptable on your last project may no longer apply, even for the same type of system.

    For example, if your project follows a risk-based approach there will be specific criteria that quality uses as a measuring stick. You may have a full suite of system lifecycle documentation, such as User Requirements, Functional Specifications, and/or a Design Specification of some sort. Depending on how the risk-based approach is structured, quality may expect your protocol to include verifications of specific elements from within these lifecycle documents. Your role in a project may be focused on protocol development, however, it’s critical that you understand validation rationales that may already be planned for your project. This is in alignment with comments by Jeff that you should familiarize yourself with quality’s expectations.

    To summarize, while it’s crucial that protocols and test cases be structured in a manner that eliminates ambiguity, understanding the rationales for testing and the appropriate deliverables that feed into the qualification effort are foundational when it comes time for a quality review.

    As a caveat, there are clearly organizations who don’t require indepth planning or advanced rationales when planning a validation effort. Whereas understanding a risk-based approach to protocol writing is critical if you want to be successful in a risk-based environment, communication of expectations is equally critical in a less rigorous business environment. In either case, the key to successful protocol development is understanding the basis that quality will be using when it’s time to review your work.

  • waynem

    Nice Post! As I work for a large company with many branches, I have had to work closely with the various Responsible Head of Quality. My experience has shown that one has to understand the Quality “Person” to be able to understand their individual needs. Some are more prone to checking the basic fundimentals such as the font,size, shades of grey, etc of the documentation while others are extremely stroppy with spelling and grammer and yet have no idea of the various tests that are to be performed. Others ignore this and show a keen interest in the why, when, and what’s of the script that one is bombarded with so many questions that one starts to question your own protocol writing skills!! lol.

    As every person has a different outlook/way of doing something such as protocols, as a Protocol writer/executioner (grin..) at least ones life will never be boring.

    Keep the posts rolling. Tx

  • waynem

    Nice Post! As I work for a large company with many branches, I have had to work closely with the various Responsible Head of Quality. My experience has shown that one has to understand the Quality “Person” to be able to understand their individual needs. Some are more prone to checking the basic fundimentals such as the font,size, shades of grey, etc of the documentation while others are extremely stroppy with spelling and grammer and yet have no idea of the various tests that are to be performed. Others ignore this and show a keen interest in the why, when, and what’s of the script that one is bombarded with so many questions that one starts to question your own protocol writing skills!! lol.

    As every person has a different outlook/way of doing something such as protocols, as a Protocol writer/executioner (grin..) at least ones life will never be boring.

    Keep the posts rolling. Tx

  • R Hess

    Interesting that there seems to be such animosity between validation teams and quality teams. I work for a very small company and the quality and validation duties overlap as far as responsible personnel.
    I have had to do much educating here on what the purpose of validations are in order to be able to work well with all departments. And truthfully, the manufacturing departments are the most resistant to performing validations.
    Maybe this is a strange question, but don’t validation teams want their documents to meet all of the quality standards too? If a question the quality reviewer brings up cannot be answered easily, then maybe the quality reviewer has a point…?
    In our company, the quality department handles most of the external audits and must assist in defending documents to customers, ISO auditors, and government auditors. As both a quality and validation position, I definetly don’t want to be caught in a position with an auditor of any kind where I cannot defend or explain any part of a validation document…

  • R Hess

    Interesting that there seems to be such animosity between validation teams and quality teams. I work for a very small company and the quality and validation duties overlap as far as responsible personnel.
    I have had to do much educating here on what the purpose of validations are in order to be able to work well with all departments. And truthfully, the manufacturing departments are the most resistant to performing validations.
    Maybe this is a strange question, but don’t validation teams want their documents to meet all of the quality standards too? If a question the quality reviewer brings up cannot be answered easily, then maybe the quality reviewer has a point…?
    In our company, the quality department handles most of the external audits and must assist in defending documents to customers, ISO auditors, and government auditors. As both a quality and validation position, I definetly don’t want to be caught in a position with an auditor of any kind where I cannot defend or explain any part of a validation document…

Similar articles:

An Alternative View of the ICH Q10 Pharmaceutical Quality System (PQS)

The image below is that depicted by the International Conference of Harmonisation (ICH) Q10, Annex 2, and is supposed to depict a PQS or Pharmaceutical Quality System.

Typically, I really love the ICH. When we have to deal with outdated regulations from different global organizations it becomes a real nightmare trying to keep track of the nuances and the ICH has done a pretty good job of bringing several of the key organizations together and aligning them on how best to organize and meet the expected requirements.

That being said the diagram below and the depiction in Q10 of what a PQS should look like is greatly lacking.

Development Phases

In section 1.8 under the Quality Manual the ICH Q10 guidance states, “The description of the PQS should include: …(c) Identification of the pharmaceutical quality system processes, as well as their sequences, linkages and interdependencies.

Process maps and flow charts can be useful tools to facilitate depicting the pharmaceutical quality system processes in a visual manner”.

I completely agree.

The problem is using the graphical depiction they present in Annex 2 is completely worthless.

Basically they listed some of the PQS elements in a bar and then said they all apply to the entire product lifecycle, which simply isn’t true.

When we are in the development phase of our product lifecycle why would we do that under the change management system, or monitor process performance?

 

Controlling Change – No Value Add

There is no point in controlling changes for a product that is purposely being changed, nor does it offer any value to monitor the process performance for a process that has yet to be developed.

This isn’t a graphic depiction of the PQS, but rather a graphic of how they depict the lifecycle management (which also has some issues).

The PQS is the quality system and its subsystems and how they interrelate.

While it’s useful to look at how the PQS and product Lifecycle Management overlap and what elements of the PQS system are relevant at each lifecycle stage, it is not the point of the PQS, and even if that’s the end goal it’s not depicted here at all.

This image offers almost no value.

A Better Approach

So, what should this graphic look like?

While this is not a perfect view of a PQS, I would propose that the image below is a much better depiction of how the PQS should be visualized and a good place to start.

At the core of any quality system should be management. This goes back to Deming, who said, “Quality begins with the intent that is fixed by Management”.

Quality has to be rooted in the executive management team.

Define Core Quality Systems

Core quality systems then need to be defined. These are systems that impact all aspects of the business and include a Risk Management Policy, Resource Management, Document Control and CAPA systems.

All of the other subsystems, Deviations, Supplier Management, Equipment Qualifications, Validation, Material Management, etc, etc. all should be risk based or involve risk assessment, they all require resources and training, they call require documents (procedures, policies, records), and the CAPA system of course drives for process improvement regardless of the process.

Subsystems

All subsystems feed back into the main Management module. The subsystems listed, all are interconnected, with the exception of Post Market Systems.

The subsystems are important too, but they are farmed out to different groups and have different levels of importance depending on the stage of the product lifecycle.

Post Market Systems

The one exception is the Post Market Systems. This includes complaint management, product reviews, recall processes and other systems to support marketed products.

These generally do not interact with the other subsystems unless it is through the CAPA system or other management functions, but still utilizes all the systems under the management umbrella.

Alternate View

The PQS presented here, isn’t intended to be perfect, but I thought it was worth presenting an alternate view to the one presented by the ICH.

The ICH concept is a good one, and the ideas are fairly well laid out in the ICH, but the graphical representation of the PQS leaves a lot to be desired.

When establishing a PQS, it is better to start with something to what we’ve depicted here, and customize it as needed for the organization.

0
shares

Similar articles:

How 21 CFR Part 11.3(7) Applies to Electronic Batch Records [Video]

When dealing with Part 11 it’s important to understand what an electronic signature actually means

The definition of electronic signatures or e-sigs can be found in 21 CFR Part 11.3(7).

Electronic Signature

An electronic signature or e-sig means a computer data compilation of any symbol or series of symbols executed, adopted, or authorized by an individual to be the legally binding equivalent of the individual’s handwritten signature.

Handwritten Signatures

We also need to understand what a handwritten signature means in the context of Part 11.
The definition of handwritten signatures can be found in 21 CFR Part 11.3(8).

Handwritten signature means the scripted name or legal mark of an individual handwritten by that individual and executed or adopted with the present intention to authenticate a writing in a permanent form.

The act of signing with a writing or marking instrument such as a pen or stylus is preserved. The scripted name or legal mark, while conventionally applied to paper, may also be applied to other devices that capture the name or mark.

Electronic Batch Records

Eric works in a Pharmaceutical company and he is responsible for the filling process of the batch been manufactured.

Each time Eric performs the filling process he has to populate a batch record with the appropriate details

After each step Eric must also fill in his signature and date to verify that he actually performed each task.

Eric is manually handwriting these details and they are legally binding to Eric.

21 CFR Part 11.3(8)

This is when 21 CFR Part 11.3(8) applies.

Fast forward 12 months and Eric’s company has implemented a brand new Manufacturing Execution System (MES) where all details around the batch manufacturing process are recorded electronically.

21 CFR Part 11.3(7)

Now when Eric performs the filling process he now populates everything electronically and signs with his username and password combination to verify that he has performed those tasks.

This is when 21 CFR Part 11.3 (7) applies.

0
shares

TOP

Similar articles: