Strapline headder

0845 345 9991

Text/HTML - benchmarks header

ODEP Benchmarks


ODEP Benchmarks


“Pre-Entry” in itself, is not a benchmark. From the beginning of ODEP, manufacturers were encouraged to register new implants with ODEP. They are asked to describe the implant in general terms and indicate where the implants are expected to be used. Pre entry gives ODEP the ability to begin tracking the implant and making sure that it continues along the ODEP pathway or is withdrawn. ODEP will contact the manufacturer after 2-3 years to invite them to submit for a level 3 benchmark (A or A*).

Because it is not a benchmark the manufacturer cannot say that ODEP have benchmarked the device and this could be important in their effort to market the product in the UK.

“Pre-Entry A* is an ODEP benchmark. It means that the implant is being monitored by “Beyond Compliance”. It will have been risk assessed by the BC advisory group and under close monitoring through the NJR by NEC Software Solutions UK (the IT contractor for NJR) with whom the manufacturer will have a contract.

The benchmarks going forward
At 3, 5, 7, 10 and 13 years there are A*, A and B ratings and these letters indicate the level of evidence that ODEP have accepted for the benchmark.

The A* level Benchmark is the highest that can be achieved at a particular time point. A summary of the requirements for an A* at each time point is shown on the grids below. There has to be a minimum number of patients that has past the benchmark timepoint and who are still alive and unrevised.

It should be noted that the acceptable revision rate for A* is the revision rate for that particular type of prosthesis plus a 2% non-inferiority margin or “cushion”. The upper 95% confidence interval for Kaplan Meier revision rate (1- Survival) must be lower than the specified level. ODEP examined revision rates for different types of joint replacement in most of the world registries that publish revision rates to determine the benchmark revision rate.

The A Level Benchmark is in many ways no less stringent but the required number past the benchmark is less and therefore the confidence intervals are typically wider. The non-inferiority margin has therefore been extended from 2% to 4%.

There needs to be the same total number in the cohort and the number of centres etc. stays the same.
The B level Benchmark is available particularly for “Boutique” products such as a DDH prosthesis and where a manufacturer has decided that they want to introduce their implant slowly into the market. The cohort size and the number at risk are much smaller and the number of centres can be no more than 1. The KM lower confidence level must be below the acceptable revision rate for that time point.

Why has ODEP decided on the time points (3, 5, 7, 10, & 13 with 15 introduced in 2020-1)?
Our philosophy is to ensure manufacturers look at the performance of their implants on a regular basis. We have taken note of the history of joint replacement failures. There have been numerous examples of implants doing well for sometimes quite a while only to fall “of the cliff” often rather dramatically.

The 3 year Benchmark
When we started NICE decreed that there should be a 3A benchmark and so we certainly started with it, only to find it worked pretty well. Generally, in the UK, surgeons are reluctant to undertake revision surgery until they are absolutely sure it is necessary and there are often issues with waiting lists. This means that at say 2 years we would be concerned that a representative “early” picture would not be seen. It is realised that statistically it would be very difficult to be sure of anything with a cohort of 150 for knees and hips and 50 for shoulders but we see the 3 level benchmark as an index of cataclysmic failure. If the size of the cohort was much larger and of size that would be sensitive to statistical analysis manufacturers might not look at their data until this large number had been achieved. That could mean that a problem did not become obvious until later than would have been ideal.

Nowadays we have Beyond Compliance in the UK and so we are in a position to closely monitor most implants in their first few years in the market.

The 5 year Benchmarks
There is no doubt that implants that are only just passing the 5 year benchmark are likely to struggle to make a 7 year benchmark. This is an early warning to manufacturers and they will doubtless assess their implant and the surgeons who are using it. This situation does not need to be allowed to fester until the 10 year benchmark.

The 7 year benchmarks
With the experience of noting several THRs going off the rails between year 5 and 10, ODEP also introduced the 7 year benchmarks in 2003-4. The 7 year benchmark is as far as a number of implants have got before they have been removed from the market. Thus we still think there is a need for a benchmark between 5 and 10 and it also helps to maintain the momentum of the process.

The 10 year Benchmarks
When ODEP was initiated by NICE the thrust was to provide patients and surgeons with a list of implants which had 10 years of data behind them with a revision rate of <10%. NICE stipulated that any implant without a 10A rating should be part of a clinical trial.

With improvement of the reported revision rates that have occurred since ODEP started it became clear in about 2012 that the 10% revision rate was no longer acceptable and that a benchmark round about 5% was where we should be. By now many of the world’s top registries had plentiful data at 10 years. At 10 years the statisticians have much more confidence in the data.

13 Year Benchmarks
With the maturity of many of the important world registries, particularly those that will cut their data at the ODEP benchmarks 13 year data is becoming increasingly available to manufacturers. Therefore, logically, it can be used to assess durability of implants in the longer term.

Obviously we have had to consider the death rate of patients with joint replacements at 13 years and in fact for the majority of implants on which we had data it was not a major problem. It may be that with implants that do not have a massive use that the manufacturer will have to wait longer to obtain the minimum number alive and unrevised at 13 years and ODEP will make allowances for that.

15 year Benchmarks
Again these benchmarks have been based on revisions rates published in international registries with a 2% cushion added.

ODEP has taken careful note of how "death" might affect manufacturers ability to source adequate cohorts. The evidence is that the majority of well performing prostheses have more than adequate cohorts to submit for a 15 year rating but there will be some that have difficulty. We have agreed that the one year's grace that is given to manufacturers to collect data for their next benchmark (before losing the benchmark), will be extended on receipt of an explanatory letter from the manufacturer. 

Example of the requirements to gain a particular benchmark, in this case Total Knee replacement (please see specific joints for their "grids").

Text/HTML - odep effects image


BC Section Menu Left


companies logos