How to handle metrics that drive the wrong behavior
March 17th, 2015 by Peter CalcottOver my career I have lived and, unfortunately, died by Metrics. What do I mean by that? Metrics if developed carefully, and with thought, can help us attain our goals. However, badly thought out metrics don’t just not help us from attaining our goals, they can actually prevent us from attaining them. Because they can be counter productive.
Is this new? No, I have seen it for decades both working in the industry but also as a consultant to the industry. I find it’s amazing that these bad metrics are not isolated to companies with poor compliance records, but also to the companies who are leaders in doing things right. Of course, with confidentiality, I can not name names, but I can talk about them and give advise in the public domain.
What are these bad metrics. They are metrics that have been set up in order to measure and control certain outputs. However, when set up, they encourage the wrong behavior. How is that possible? I will give you two examples. Now, let me explain.
- First example
I was visiting a company recently for a training session on quality systems. During the break, one of the attendees took me aside and described a situation at his company. He asked the situation to be kept private. And by that he meant that not only should I not describe it in public describing his company but he did not want management at his company to hear about it ascribed to him. Of course, I honored his request so you will learn neither who the person is or the company.
As with most companies, they wrestle with investigations taking long times to close out which leaves the company vulnerable both operationally and from a compliance perspective. So to combat that and to drive closure, the company instituted a metric of “All investigations to be closed in 30 days”. Depending on the number open during the year, the persons performance rating would be impacted. Performance impacted equals decreased pay raise, bonus etc. You get the picture.
Result, the number of investigations lingering past 30 days goes down. Management is content and everything is improved. Or is it?
The answer is no!!!!! Yes, investigations are closed out quickly, but are they really completed and accurately describe what the root cause or contributing factors are? In the haste to get a good grade, people are closing out investigations prematurely with poor root cause analysis. Without this “good” investigation, the CAPAs developed are not directed to the right things. So the CAPAs do not solve the problem and the result is that the problem reappears – in other word, we get repeat observations.
A better metric would be a goal of no repeat deviations or discrepancies. That would indicate the CAPA worked because the investigation was thorough. With decreased repeats, the work load would decrease giving better opportunity for effort on the unique observations.
2. Second example
I was visiting a client one day to examine their quality systems, especially deviations and their handling. I had flown for several hours to visit the company and was met by the plant manager who indicated that the issue that I was there for had been solved and they did not need my services for that. Since I was already there, and he was paying anyway, I suggested I look at the remediation and maybe other systems in need of help. So off we went.
Apparently, over the last few weeks the PM had had a great idea. He linked pay for performance to the number of deviations in the department. And immediately (for the last two weeks at least), there had been a 20% reduction in deviations. The first thing I did was to go to the lot disposition department to see how things were and talked to the staff there. They immediately reported that over the last week or so there had been an increase in the number of batch records arriving with serious errors and deviations that had not been highlighted. Previously, the Production department were encouraged to self report deviations and highlight them to QA. Now it was up to QA to try and find the errors. Clearly a step backwards. So the metric of reducing deviations had not decreased the number but rather the reporting of the deviations. The deviations were still there but not reported. We all want less deviations but this is not how to get it.
In both these incidents, the metric had driven the wrong behavior. So how do you set up metrics that work. I recommend this simple process.
- First identify the system that you want to work on. In these cases, the investigation system.
- Define the out come you want.
- In the first, closure of investigations. While timeliness is important, surely, getting it right so we have a good chance of an effective CAPA to prevent recurrence is the real goal.
- In the second case, of course, you want to get no deviations, but if they have occurred, you want them reported, so they can be investigated properly so we can get effective CAPAs so they don’t appear again.
- Based on the input of point 2, set up metrics to drive the right behavior.
- For example one, you can have a metric of no repeat observations. That indicates that the investigation was thorough and the CAPA directed to the right thing. Hence, it solves the problem.
- For example 2, we want batch records to arrive in QA – right first time (RFT). That is completed, checked and all deviations highlighted and put into the system for resolution.
Both these sets of metrics looks in to the future versus simply the immediate.
Are these the only examples or areas. Of course not, but if you follow these principles, you will get improved operations. Before any metric is established, ask the questions”will this metric drive the behavior and result I really want?” And be careful what you ask for. It might not be what you really want.