DIA Metrics Session Provides Guidance for Sponsors and Vendors: A report on a key industry topic from the 2009 DIA Annual Meeting

Metrics was a topic on many attendees’ mind at the 2009 DIA meeting in San Diego. Sponsor organizations, CROs and EDC companies alike have recognized the need for metrics to help differentiate and to accurately measure their performance. Over the last few years, companies have worked together through an industry consortium to draft a set of consistent metrics to evaluate the relationship between sponsors and outsourced service providers.

Representatives from Metrics Champion Consortium (MCC) spoke to hundreds of DIA attendees about metrics and the future for measuring and evaluating performance – both on the part of the drug/device company, as well as the vendor. MCC is a non-profit organization of biotechnology, pharmaceutical and service provider organizations. Their mission is to “develop, through a collaborative process, Performance Metrics within the Biotechnology and Pharmaceutical industry with the intent to jointly encourage performance improvement, effectiveness, efficiency and appropriate levels of controls for both Sponsors and Service Providers in support of the drug development process.”

One purpose of metrics is to make communication between sponsors and vendors more productive and beneficial to the business exchange. With millions of dollars and years of work at stake, everyone in the drug industry is looking for proof points for every decision they make, especially when sponsors are starting a new study and looking for a new service provider. For vendors, metrics indicate areas for improvement and proof points to advocate for their particular approach to clinical outsourcing. The risks of wasted time, money and resources are significant and the use of metrics aims to reduce risk by increasing efficiency through timely measurement of key activities and relationships.

Through comprehensive, accurate metrics, sponsors and vendors can more easily meet these overall goals:

  • achieve a clear picture of both parties’ capabilities to handle a trial
  • cultivate constructive conversation around specific trial aspects
  • build and reflect the nature of the relationship between sponsors and vendors

The MCC made a list of five working groups to classify types of metrics: Biometrics, Business Operations, Clinical Operations (Clin Ops), Drug Supply, and QA / Regulatory / Safety. These core competencies guide the creation of metrics. The MCC’s process for creating a clinical trial performance metric is essentially answering twelve components. The answers provide all the relevant information about the discovery, method, implantation and protocol for each metric.

The MCC formats the resulting conclusions:

Once a metric is defined and calculated, it is categorized as either a leading indicator or a lagging indicator. Leading indicators show opportunity for change within a current trial based on that reported metric. They are predictive and can provide forward-looking glimpses into the progression of a trial. For example, the measurement of time between protocol approval and first patient, first visit (FPFV) for all sites might indicate how the project will track to plan and could indicate the need of a recruitment risk strategy.
Lagging indicators show opportunity for change in a future trial based on results of previous trials. They are statements of what has already occurred, and are best looked at to evaluate performance for future work. They are results instead of a prediction. For instance, a metric around the time to database lock is of value, but only for looking at past performance, not for determining corrective action for the existing trial.

Since each trial is unique and requires special attention, metrics addressing a sponsor’s concerns provide the relevant information on which to base contract decisions. Costs, timelines, quality, or reliability are a few key performance indicators (KPI) of main concerns for sponsors and vendors. However there are also qualitative metrics.  Key Relationship Indicators (KRI) serve the same purpose of KPIs but for less quantitative factors.  A KRI may measure project management communication, site satisfaction with support, or other qualitative measures. 

Qualitative or quantitative metrics chosen with care will yield a clearer picture of what both the sponsor and the vendor can accomplish together for a study. It is important to note that the use of clinical trial metrics is to evaluate both the sponsor and the vendor effectiveness. The use of service level agreements in contracts needs to be measured by both sponsor and vendor hitting their key performance indicators. One such measurement is around timelines.
Timeliness is crucial for a successful study. It can be measured for Clin Ops and overall project management. One measurement for an overview can show the actual time vs. budgeted time vs. work completed in that time. Or more specifically, from a Clin Ops perspective, a measurement of the time from signing of the investigator contract to FPI at a site can indicate efficiencies. From a project management perspective, it could be indicative to measure the time lapse from protocol approval at FPI, to the last subject out, to the database locked. Everything from shipment delays to efficient data entry can be measured and utilized as a metric for validation of use or denial of outsourcing.

Some companies and technologies make tracking metrics even easier. Datatrial is a boutique clinical data organization that offers an array of services from data management and statistics to its own electronic data capture (EDC) software, nowEDC™. This sophisticated software, accessible from any PC, is easy to learn and manipulate to meet specific requirements and allows for a rapid development time to get from user acceptance testing to FPI. The use of EDC allows for many key performance indicators, such as the time between a patient visit and the entry of that data on an eCRF. Site payments can be triggered off such metrics, so there is an element of performance-based pay introduced on clinical trials. These type of metrics highlight the value of using nowEDC™ software or other industry-leading technology.

Metrics tracking clinical trial elements from cost efficiencies to timelines aim to identify what works, what doesn’t work, and how to drive appropriate changes in drug and device research. CROs and EDC providers should develop metrics to demonstrate their value and core competencies to sponsors; sponsors should request specific metrics so they will have hard data that can help them make informed decisions about how best to get drugs to market.

While advances have been made, there is still far to go to create industry-wide metric standardization.  Before this can be reached, the MCC wants to gather forces from all across the industry to collaborate in generating the ideal set of metrics for every company to deliver.  The next step is for everyone is to get involved.  The MCC requests feedback on the process for creating a metric, and the types of performance indicators and relationship indicators that are most important.  Their site, https://www.metricschampion.org contains links to forms for feedback on their presentations, as well as information and email addresses to communicate specific thoughts. Supporting the cause of metrics will enable the biotechnology and pharmaceutical industries to increase efficiency and communication in order to reach an overall industry goal: longer, healthier lives.

    PowerPoint presentation by Cory Gutterman: MCC CRO Initiative Working Group Kick-Off Session.
    PowerPoint presentation: MCC Clinical Trial Performance Metrics Industry Feedback Process
    DIA 2009 Session: Utilize Standardized CRO Performance Metrics to Drive Appropriate Change