Key Risk Indicators (KRIs): your backbone for operational success

A well-defined KRI process can have huge impacts on clinical trial quality and efficiency.

According to the Transcelerate definition "A Risk Indicator can be qualitative (for example: a site monitor’s assessment of site quality) or quantitative information that is used to monitor identified risk exposures over time, and are in many cases determined by comparing across programs, protocols, countries or sites, and are predominantly used at the patient level."

Transcelerate also includes a definition of thresholds: "Thresholds are the level, point, or value associated with a Risk Indicator that will trigger an action."

...I agree with the definition, though I'd like to challenge study teams to reduce the qualitative KRIs or risk indicators as much as possible. Even if a "site quality finding" is qualitative when described it can be helpful to define a KRI of "1 or more qualitative findings" to trigger an action, for example. This also helps satisfy the need to define a threshold.

Risks vs. KRIs:

Risks are typically expressed in more general terms, e.g. "We have a concern over high dropout rate due to the treatment burden over time". Expressing risks in easy-to-consume statements will allow for smoother discussion and agreement from the study team. I explore the risk assessment and definition process in this post.

A KRI is the pre-defined signal of the risk potentially occurring, resulting in monitoring and action. For example:

1. "Dropout rate >10% by the week 4 visit" could be a KRI providing an early signal of a high dropout rate for the study.

   1. A 30% overall dropout rate could be the Quality Tolerance Limit, at which point the Critical-to-Quality factor for high dropout has been realized (which then negatively impacts the reliability of treatment analysis).

2. When the 10% KRI threshold is met, the team will take action to try and address the risk well before it reaches the Quality Tolerance Limit of 30%.

   
   

...So the ideais to implement a KRI that helps the team flag an early trend of dropouts, perhaps based off a known risk that participants often dropout at or near 4 weeks on treatment.

Keys to successful KRIs:

• Specific, and based on risk statements: risk statements should be defined as part of your overall risk management process per trial. Your KRI supports your risk statement as mentioned above.

  
  

• Data-driven: a metric and threshold that triggers action, even if that action is further discussion and review.

  • E.g. ">3 participants have SAEs related to abdominal discomfort" -if this is at a specific site it may trigger a safety call between the Principal Investigator and Medical Monitor.

    
    

• Proactive: KRIs should be "warning signals" that help address issues at the earliest sign of needed action, then potentially revised once the threshold is met.

  • E.g. "participant adherence to the daily questionnaire is <80% week-on-week" focuses on a weekly rate to support earlier action.

  • E.g. ">5 sites with instrument breakdown" may need to be updated to ">10 sites..." if the the first threshold is realized.

    
    

• Timely: KRIs need to be delivered based on urgency.

  • They could be output via real-time notifications (high urgency), periodic reports (urgency depending on frequency), dashboard metrics, study team calls or during periodic project reviews (less urgent).

    
    

• Actionable: there needs to be clarity on what action to take, therefore it is important that the data, report, visualization etc. delivers clear and concise information that the recipient knows what to do with. It is also helpful to define an action associated with the KRI, even if it is a general guideline.

  • Knowing there are 200 queries outstanding is helpful. If 150 of those queries belong to 1 sites the data is then actionable.

    
    

• Ownership: notifications or reports that land in the email ether are of no help to anyone. It is important to identify an owner for the KRI, as well as a backup. And it makes sense to have the owner be a stakeholder, for instance someone who's job or focus is related to the KRI.

KRIs can cover the following, and more:

• Early indicators for Quality Tolerance Limits related to Critical-to-Quality factors: since QTLs indicate that a CtQ factor is "at risk", KRIs can be defined as early indicators to help proactively mitigate a QTL that may be on the horizon.

  
  

• Efficacy: metrics or data expected for proper study analysis, e.g. primary endpoint assessment.

  
  

• Safety: serious, expected or unexpected AEs that warrant comprehensive evaluation; special lab values and alert ranges, specific vital signs results or physical examination findings, patient reported suicidal ideation, drug accountability...

  
  

• Study performance: enrollment tracking, early termination trends, outlier trends across participants, country level trends...

  
  

• Site performance: tracking eLearning completion, access to key portals, outstanding queries, query response rate, randomization and key visit or assessment completion...

  
  

• Participant performance: number of unscheduled visits, time on task for PROs, missing or incomplete patient reported information, patient devices offline, early monitoring for initial study success, early termination reasons...

  
  

• Data integrity: missing critical data points, unexpected results for important assessments, missing data thresholds, high query rates for specific fields...

  
  

• Early warning: data that provides early indication of future study challenges, e.g. during screening. Signals or indicators could include incomplete tasks, high time-on-task or highly variable completion times.

Here are a few articles covering explanations and examples of KRIs:

• Transcelerate introduction

• Applied Clinical Trials, 2016

So how can the use of KRIs impact the operational delivery of your clinical trial with such significance?

We know that high levels of quality should be the focus of a good clinical trial. But quality is inherently at risk when there are variables, known and unknown, that can impact the deliverables which comprise quality. While we hope these risks are unlikely, it is important to assess and understand them. And if there are important risks that could impact the quality outcomes (CtQ factors and beyond), we should be monitoring and actioning those risks via KRIs before they result in negative or less-than-ideal outcomes of the study's quality.

In other words, don't put quality at risk!

Back to the table of contents.