Text/HTML-knee0head

ODEP Methodology

Text/HTML


ODEP Methodology For Shoulders - Frequently Asked Questions


1. Why are the criteria for benchmarking in shoulders being updated?

2. How will the criteria be applied?

3. Why is the minimum total cohort used for ratings?

4. What is the minimum at risk at benchmark time?

5. How are the maximum revision rates criterion established?

6. Why are PROMs being asked for A* ratings?

7. What PROMs will be accepted and on how many patients?

8. How will PROMs be applied for A and B ratings?

9. Will currently published implant ratings still be recognised?



1. Why are the criteria for benchmarking in shoulders being updated?


ODEP has been rating shoulder replacements since 2017 based on data submitted by manufacturers and has used an early version of ODEP for shoulders submissions grid to rate these implants. ODEP for shoulders is in evolution and has benefitting from the data collected over the last three years and the involvement of all stakeholders agree that an update to the standard benchmarking criteria for shoulders is required.

These will be used to review the future submissions from summer of 2021 and will be appraised every three years based on evolving evidence, data available from joint registries and feedback from all stakeholders.

This is in line with the recommendations from the publication of the Cumberlege report of the Independent Medicines and Medical Devices Safety Review in July 2020.


2. How will the criteria be applied?

The new criteria for shoulders have been developed through a combination of using published evidence on revision rates for shoulder replacement, National Joint Registry outcomes data and a consensus exercise of the ODEP for Shoulders members.

These criteria are already being submitted by a number of manufacturers and will be applied to assess future submissions for rating of shoulder replacements, once all the involved stakeholders are confident that this method will deliver an enhanced, robust and reliable system to rate shoulder replacements.

 

The companies will have to agree to a declaration in the clinical data template that the data submitted for this product is representative of the results of all studies or other clinical studies available. Details will need to be provided of, and reasons for, data that has been omitted that does not indicate similar or better results than that submitted on the clinical data template.

 

3. Why is the minimum total cohort used for ratings?

To assure the public that an implant is safe to be used in the long term, outcome data needs to be available on a minimum cohort of joint replacements to offer an appropriate degree of certainty that the product is safe and is performing well in the long term. For a more precise estimate of performance to be calculated, larger numbers are required to provide a high degree of certainty, which means exposing larger numbers of patients to the new product, which the panel do not consider safe.

The suggested minimum total cohort for each set of ratings are based upon NJR reports, where sufficient data is available for a number of implants from different manufacturers to measure the cumulative survival ratio for survivorship analysis for revision. This data can be requested from the NJR for future submissions for the ODEP ratings at the different benchmark times.


ODEP will continue to accept aggregated data from more than one source to produce a cohort size for a particular benchmark. However, it must be clear that the data from different sources, are not using the same patients, as this would be duplication.


4. What is the minimum at risk at benchmark time?

At each benchmark time, ODEP uses the Kaplan Meier survival analysis to demonstrate (with 95% confidence) that the underlying revision rate for a brand is better than a specified level. For this calculation, the minimum total cohort and a minimum patients at risk at each benchmark time is required. Thus at each bench mark time point a total cohort number and a minimum number of patients still at risk at that time point has been specified that is lower than the total cohort to allow for losses due to death or follow-up. This number is based on the numbers available in the NJR for the different brands from major implant companies and the recommended minimum numbers for robust statistical calculations to provide confidence in the data.

For A ratings, the required number past the benchmark has been reduced to the bare minimum necessary to avoid anomalies where there are zero revisions. It has been calculated so that, if there are zero revisions, the upper confidence limit for the binomial exact confidence interval for the proportion revised will be lower than the benchmark maximum revision rate. This is necessary since the standard Greenwood method cannot calculate a KM confidence interval where there are zero revisions.


5. How are the maximum revision rates criterion established?

The maximum revision rates criteria for shoulders have been established using a combination of published evidence on revision rates for shoulder replacements, National Joint Registry (NJR) outcomes data published in 2020 and a consensus exercise of the ODEP for shoulders members. Following an in-depth analysis of registry data from other countries, we have found that the maximum revision rate is approximately 1.5%-2% higher than the overall revision rates published in the NJR hence we feel that the 2% cushion is a workable way forward.

For benchmarking at each timepoint, data would need to be submitted for a cohort demonstrating revision rates at the benchmark times and Kaplan Meier survivor implant data showing upper and lower confidence limits on the data.

For A* ratings, the upper 95% confidence interval for KM maximum revision rate must be lower than the specified level at each time point. The criterion for the maximum revision rate is the same for the A ratings but the PROMs submission is not essential.

For B rating, the criterion for maximum revision rate remain the same but maximum value of 95% lower confidence limit for revision rate at the particular timepoint must be below the specified maximum revision rate.

 

 

Text/HTML - odep effects image

Example ratings for Shoulders

 

Mark
applied
for
Total
cohort
Number
at risk
Revision Rate
95% CI
PROMs
available
Rating Reason
3 140 100 2.58% - 4.71% Yes 3A* Meets all benchmark criteria
5 180 130 4.75% - 6.50% Yes 5A*  Meets all benchmark criteria
7 240 170 6.52% - 8.25% No 7A Insufficient or no PROMs
10 250 140 6.04% - 9.03% No 10A Insufficient numbers for A*
5 150 120 3.07% - 7.37% No 5B Upper CI > 7%
3 100 70 5.38% - 10.24% No U Lower CI > 5.0%

 


6. Why are PROMs being asked for A* ratings?


Revision is not a reliable end point with shoulder replacements as in some patients whose shoulder replacements have failed, it is sometimes chosen not to revise the joint due to risks associated with further surgery and lack of suitable surgical options. PROMS data may be able to discriminate between better performing implants. Whereas with hips and knees, revision is generally accepted as an acceptable end point, with shoulder replacements this is not always true. 

In addition, the Cumberlege report 2020 has emphasised the importance of PROMs in monitoring of all medical devices. The report recommends that a central patient-identifiable database be created by collecting key details of the implantation of all devices at the time of the operation and then be linked to specifically created registers to research and audit the outcomes, in terms of both the device safety and patient reported outcomes measures. We support any independent method of PROMs collection for shoulder arthroplasty, including online portals and/or manual collection by allied healthcare professionals.


For A* ratings for shoulder replacements, the panel will accept all validated shoulder specific outcome measures e.g. Oxford Shoulder Score, American Shoulder & Elbow Score, Constant score, DASH score or any other validated score.

It is also desirable that preoperative and postoperative benchmark PROMs data is submitted on at least 50 patients is included in the submission. If preoperative data is not available, then 6 month postoperative PROM outcome score could be used as a baseline to determine the change in score at each benchmark time. For example, if an application has been submitted for a 5 year A* rating, preferably preoperative or 6 month post-surgery PROMs data and the corresponding PROMs score at 5 year on at least 50 patients of total cohort of patients will need to be submitted. 

PROMs and other data does not need to be derived from the UK NJR and indeed, manufacturers are encouraged to submit PROMs and other data from other sources, including alternative, national registries.  ODEP has always accepted that manufacturers can aggregate data from more than one source to produce a cohort size for a particular benchmark, but using data from more than one source, where in fact the patients are the same, is not acceptable. The companies will have to agree to a declaration in the clinical data template that the data submitted for this product is representative of the results of all studies or other clinical studies available. 


If it not possible to provide a linked cohort with 30% of baseline and PROMs at the appropriate timepoint being rated then the panel will still consider the submission for A and B ratings, and so the panel encourage industry to submit with as many linked PROMs as possible, to achieve the higher ratings.


The ODEP shoulder members would like to reassure manufacturers that all previous ratings will continue to be recognised and published on the website according to the previous guidelines.

The inclusion of the A* ratings is an opportunity to highlight the best performing implants, which we have not been able to provide manufacturers the mechanism to do so previously.

 

 



23 November 2020

Revised June 2021


BC Section Menu Left

Menu

Text/HTML