The Calcott Consulting Blog:

   Articles on Contract Manufacturing:


Effective auditing – one size does not fit all

April 15th, 2015 by

Every company I work with has a problem with their auditing program.  Some believe they are overdoing it and others feel they are not doing enough.  In a sense, they are both right.  In actual fact, they are all overdoing it in certain areas and underdoing in others.

Back when I started in Industry, and I am afraid to tell you exactly when, auditing programs were written in stone in SOPs.  The frequency, duration and number of people were defined and rigorously enforced.  Audits were conducted each the same way and lists of findings were assembled and sent to the auditee.  If lucky, the findings were responded to and CAPAs developed.  The report was closed out and filed.  After the prescribed period of time, the process was repeated.  Often the same findings were seen at the next audit.  So either the CAPA was not done or it was ineffective.  Not exactly an efficient, effective process, but it satisfied the regulators.

Today a program like that is just not acceptable.  Why?  Have the regulations changed? Have our expectations changed?  Has the world changed?  The answer to each is yes.

Over the last 20-30 years we have seen a lot change.  We have seen drug tampering (Tylenol and cyanide), counterfeit drugs in the market place (you get those emails for those drugs at unbelievable prices) and incidents like the Heparin / Baxter problem.  Both industry and regulators have taken note and reacted.  In the US and EU, regulators have recognized the problems and issued new regulations and guidances.  The Falsified Medicines Directive (FMD), Food and Drug Administration Security and Innovation Act (FDAsia) and the Drug Supply Chain Security Act (DSCSA) have been issued and are in the process of implementation.  So how does that fit into the auditing program?  It is because the auditing program is a tool that will enable you to meet the spirit of what these regulations are driving at.

We perform both internal audits of our own operations as well as audits of third parties.  These third parties include our CMOs, our suppliers of raw materials, excipients, actives as well as services such as testing labs, engineering functions and distribution to name a few.  The functions of an audit are many fold including a component of the assessment of whether we care to do business with an entity (the Vendor Qualification Program) as well as as a routine assessment of whether we want to continue to use them (continuous verification) and an assessment after some element has failed (for cause).

Each of these is approached differently, depending on the nature of why we are auditing.

  1. Vendor assessment – usually, you have never worked with the vendor before, or at least recently, so this is an exploratory audit to assure they are operating to an appropriate standard that is compatible with our expectations.  Because of this, the goal is to assess all their systems.
  2. Continuous verification – you have experience with this type of vendor.  You know what they do well and perhaps have identified areas where improvement might be needed.  You are often following up from previous audits or experience with their services (described in the annual product review).  So it is often more directed than the qualification stage of vendor assessment.
  3. For cause – something has definitely gone wrong.  So this is a very directed audit towards the areas of potential deficiency.  The outcome may be to continue to use or to terminate the relationship.

Which brings us to how to conduct an audit to add value.  ICH Q9 is a wonderful guidance that if used intelligently can aid you in developing a truly risk-based auditing program: that is to balance the “too much” versus the “too little”.  I highly recommend integrating this guidance into your auditing program.  Remember if you do not document your risk decisions, you will be found lacking by the regulators.

I use the old moniker of

say what you do,

                    do what you say,

                                          prove it and

                                                           improve it.

Put another way it is really documentsexecutionrecords and continuous improvement.

These following steps may aid you in defining the audit program.

  1. Never schedule your auditors more than 67% of their time and that includes prep time and report writing time.  The extra 1/3 is important for the unexpected such as the for cause audits, the new suppliers, the new emergency programs and also the deep dives you provide as a service to your internal customers.
  2. Determine the risk factor for the particular vendor.  That includes not just the service provided but the track record of each.  This will determine the frequency, duration and manpower needed. And this needs to be kept current because situations change.
  3. You have limited time at the vendor so use it well.  Prepare the outline of what you want to accomplish (the type of audit), what you know, what questions need answering.  If possible do work before you arrive.  That could be sending out a focused questionnaire to relatively simple elements (you can confirm when you arrive).  Even present to them a proposed agenda, so they are prepared and have no excuse when you arrive.
  4. So what do I focus on when I arrive.  A typical process flow approach is my choice.  For actives suppliers or CMOs, I walk the process with my questions and get my answers in situ.  Armed with my preparation work, I walk through the facility and quite prepared to stop even for a significant length of time to explore more if I sense an issue.  For testing labs walk the samples.  This is the execution component
  5. I look at paper work later and I focus on the various quality systems of interest.  I do not read SOPs or policies but rather focus on the records part.  I look at deviations and investigations, CAPAs, change controls and lot dispositions.  The threads I find lead me into the various other systems.  I find these systems are the pulse of the organization and tell you a lot about the company.
  6. If necessary, I go to SOPs and policies.  That is the documentation part to confirm that what they say corresponds to what they do.
  7. I also look at operations and people to detect signs of continuous improvement which is often picked up, not in documents, but in conversation.
  8. I usually look to see evidence of a modern approach to quality as evidenced by an active involvement of management.
  9. In the close out I gather the observations which I have ranked using the EU standard of critical, major and minor.  In the discussion I might even make some suggestions of how improvement might be made.  But it is the company’s decision on the how to address really.
  10. After you get home, make sure to follow up with requests for CAPAs after the agreed upon time frame.

BTW, one of my first stops is the bath room.  Not because of a medical problem but  to see how it is kept.  Companies that have a good QMS have clear bathrooms.  For those with QMS problems, the bathroom can be a telltale.

How to handle metrics that drive the wrong behavior

March 17th, 2015 by

Over my career I have lived and, unfortunately, died by Metrics.  What do I mean by that?  Metrics if developed carefully, and with thought, can help us attain our goals.  However, badly thought out metrics don’t just not help us from attaining our goals, they can actually prevent us from attaining them.  Because they can be counter productive.

Is this new?  No, I have seen it for decades both working in the industry but also as a consultant to the industry.  I find it’s amazing that these bad metrics are not isolated to companies with poor compliance records, but also to the companies who are leaders in doing things right.  Of course, with confidentiality, I can not name names, but I can talk about them and give advise in the public domain.

What are these bad metrics.  They are metrics that have been set up in order to measure and control certain outputs.  However, when set up, they encourage the wrong behavior.  How is that possible?    I will give you two examples.  Now, let me explain.

  1. First example

I was visiting a company recently for a training session on quality systems.  During the break, one of the attendees took me aside and described a situation at his company.  He asked the situation to be kept private.  And by that he meant that not only should I not describe it in  public describing his company but he did not want management at his company to hear about it ascribed to him.  Of course, I honored his request so you will learn neither who the person is or the company.

As with most companies, they wrestle with investigations taking long times to close out which leaves the company vulnerable both operationally and from a compliance perspective.  So to combat that and to drive closure, the company instituted a metric of “All investigations to be closed in 30 days”.  Depending on the number open during the year, the persons performance rating would be impacted.  Performance impacted equals decreased pay raise, bonus etc.  You get the picture.

Result, the number of investigations lingering past 30 days goes down.  Management is content and everything is improved.  Or is it?

The answer is no!!!!!  Yes, investigations are closed out quickly, but are they really completed and accurately describe what the root cause or contributing factors are?  In the haste to get a good grade, people are closing out investigations prematurely with poor root cause analysis.  Without this “good” investigation, the CAPAs developed are not directed to the right things.  So the CAPAs do not solve the problem and the result is that the problem reappears – in other word, we get repeat observations.

A better metric would be a goal of no repeat deviations or discrepancies.  That would indicate the CAPA worked because the investigation was thorough.  With decreased repeats, the work load would decrease giving better opportunity for effort on the unique observations.

2.  Second example

I was visiting a client one day to examine their quality systems, especially deviations and their handling.  I had flown for several hours to visit the company and was met by the plant manager who indicated that the issue that I was there for had been solved and they did not need my services for that.  Since I was already there, and he was paying anyway, I suggested I look at the remediation and maybe other systems in need of help. So off we went.

Apparently, over the last few weeks the PM had had a great idea.  He linked pay for performance to the number of deviations in  the department.  And immediately (for the last two weeks at least), there had been a 20% reduction in deviations.  The first thing I did was to go to the lot disposition department to see how things were and talked to the staff there.  They immediately reported that over the last week or so there had been an increase in the number of batch records arriving with serious errors and deviations that had not been highlighted. Previously, the Production department were encouraged to self report deviations and highlight them to QA. Now it was up to QA to try and find the errors.  Clearly a step backwards.  So the metric of reducing deviations had not decreased the number but rather the reporting of the deviations.  The deviations were still there but not reported.  We all want less deviations but this is not how to get it.

In both these incidents, the metric had driven the wrong behavior.  So how do you set up metrics that work.  I recommend this simple process.

  1. First identify the system that you want to work on. In these cases, the investigation system.
  2. Define the out come you want.
    1. In the first, closure of investigations.  While timeliness is important, surely, getting it right so we have a good chance of an effective CAPA to prevent recurrence is the real goal.
    2. In the second case, of course, you want to get no deviations, but if they have occurred, you want them reported, so they can be investigated properly so we can get effective CAPAs so they don’t appear again.
  3. Based on the input of point 2, set up metrics to drive the right behavior.
    1. For example one, you can have a metric of no repeat observations.  That indicates that the investigation was thorough and the CAPA directed to the right thing.  Hence, it solves the problem.
    2. For example 2, we want batch records to arrive in QA – right first time (RFT).  That is completed, checked and all deviations highlighted and put into the system for resolution.

Both these sets of metrics looks in to the future versus simply the immediate.

Are these the only examples or areas.  Of course not, but if you follow these principles, you will get improved operations.  Before any metric is established, ask the questions”will this metric drive the behavior and result I really want?”  And be careful what you ask for.  It might not be what you really want.

Putting Quality Agreements into practice

June 24th, 2014 by

It’s been about 1 year since the FDA issued its Guidance on Contract Manufacturing.  Based on what I read in blogs and social media like LinkedIn, I am sure not all companies are really up to speed. At an IBC conference recently, I presented a paper on the guidance and putting it into practice.  It was so well received, I was asked to write an article on the topic by Bioprocessing International and here it is.  Good reading!!

What does a Quality or technical agreement have to contain?

September 19th, 2011 by

I was on a panel discussion on Contract Manufacturing in Berlin this last week and the topic of Quality or Technical Agreements produced a vigorous debate. While everybody agreed that the requirements for the documents are codified in the EU by regulations, the FDA does not directly require them by law. However, as most know, if you do not have them in place, you will get a tough time at inspection to assure that you have appropriate oversight of your Contract Manufacturer (CMO).

So what should you have in the document and how should it be controlled?

Firstly, essentially everybody at the session was adamant that the document was one whose purpose was to define expectations between both parties. Again essentially all believed it was a document written by the Quality group to define the quality expectations, obviously involving other disciplines eg, manufacturing, logistics etc for input.

However, who should have control of the document split the group into two camps. The majority believed, that these parties alone needed to review and approve the document. A small minority (including the only lawyer present) believed it should be approved by the legal function, and be formatted by legal. I tend to believe that the value of legal should be advisory in the preparation to assure that no landmines or bombshells are present.

However, it is a document used by both quality and other technical functions. As such it needs to be written in real language that is understood by the doers. So in my experience, I have involved the legal department for advice, which I do take seriously. But I do not let it get hung up in the legal department in their processes which tend to take extraordinarily long timeframes to complete. Usually, I give then a week or so to review with the comment that no communication by the end of the week indicates no problems found. Of course, I do not wait until the department is involved and tied up in some busy critical project or most are on holiday. Rather I plan ahead giving them a heads up that one is coming along. In other words, I play a balanced hand but do not let them become either the gatekeeper or the bottleneck.

That said: what should it contain? The simple answer is “everything that appears relevant”.

Clearly, define the product(s) and steps in manufacture that are covered by the relationship and the markets that will be supplied. This is to assure that the correct GMP’s will be applied.

With this as a preamble, go through each of the quality systems, to assign responsibility to assure who will do what is clearly defined. In some cases, one party will be responsible for all activities. But in most cases while one party will be responsible for most, the other party will play a role even if it is simply to be informed of the result.

So why am I not describing who is responsible, system-by-system? It is because there is no set pattern and it must be defined for your particular circumstance. Who does what is determined by who has the expertise and the risk level you are prepared to tolerate.

One mistake people make is to assume that if I am outsourcing the physical manufacture, I can also outsource the responsibility and the accountability. Truly delegate as much as you can to your CMO. You are paying a highly experienced entity: so use them. However at the end if the day you are responsible for the activities at the CMO. Make sure you are involved in change control, lot disposition, investigations (major or critical), just to name a few.

Communication is critical. Define the who, when, what, where and why. Make sure the contacts are clear to both parties and their communication channels are open. In the world of improved e-communication, while email is exceedingly good, there are times when face to face is critical. Frequency of meetings and reports are important to build expectations. Clearly defined mechanism prevent the data overload of random people called random people in the other company.

Sometimes things go wrong and an audit is need for cause. Define the routine and non routine so it is clear when you can go. Include in that you can use consultants to audit. Put in a section on what to do when you butt heads. You hope you ever need it but if you do, you have it spelled out.

Change controls can be simple for small plant-only activities. It can be more complex if it results in a submission. In the latter cases, make sure you are an active player. Make sure you keep up to date so you can submit supplements quickly and on time. You are releasing product to market, so you must be involved in the operations. You need to know all the issues with the lot and make sure the contractor communicates to you when things go wrong. Definitely be fully aware of all critical deviations or discrepancies including OOS’s.

In general terms, activities that are truly plant specific can be delegated to the CMO with you having the option of auditing periodically or receiving monthly reports. For product specific activities, I recommend you taking a leadership role approving all activities. This applies to validation in particular.

I periodically present webinars on this and other topics in conjunction with Tungsten Shield. Of course a 2 hour webinar will cover much more detail that that described here. Check out their site for upcoming webinars I will be presenting.