ODEP Benchmarks Explained
“Pre-Entry” in itself, is not a benchmark. From the beginning of ODEP, manufacturers were encouraged to register new implants with ODEP. They are asked to describe the implant in general terms and indicate where the implants are expected to be used.
ODEP then have the implant on their books and can contact the manufacturer after 2-3 years to invite them to submit for a level 3 benchmark (A or A*)
Because it is not a benchmark the manufacturer cannot say that ODEP have benchmarked the device and this could be important in their effort to market the product in the UK
Pre-Entry A* is an ODEP benchmark. It means that the implant is being monitored by “Beyond Compliance”. It will have been risk assessed by the BC advisory group and under close monitoring through the NJR by Northgate PS (the IT contractor for NJR) with whom the manufacturer will have a contract.
The benchmarks going forward.
At 3, 5, 7,10 and 13 years (soon 15 year benchmarks will be added) there are A*, A and B ratings and these letters indicate the level of evidence that ODEP have accepted for the benchmark.
The A* level Benchmark is the highest that can be achieved at a particular time point. A summary of the requirements for an A* at each time point is shown on the grids below. There has to be a minimum number of patients that has past the benchmark timepoint and who are still alive and unrevised
It should be noted that the acceptable revision rate for A* is the revision rate for that particular type of prosthesis plus a 2% non-inferiority margin or “cushion”. The upper 95% confidence interval for Kaplan Meier revision rate (1- Survival) must be lower than the specified level. ODEP examined revision rates for different types of joint replacement in most of the world registries that publish revision rates to determine the benchmark revision rate.
The A Level Benchmark is in many ways no less stringent but the required number past the benchmark is less and therefore the confidence intervals are typically wider. The non-inferiority margin has therefore been extended from 2% to 4%.
There needs to be the same total number in the cohort and the number of centres etc. stays the same
The B level Benchmark is available particularly for “Boutique” products such as a DDH prosthesis and where a manufacturer has decided that they want to introduce their implant slowly into the market. The cohort size and the number at risk are much smaller and the number of centres can be no more than 1. The KM lower confidence level must be below the acceptable revision rate for that time point.
Why has ODEP decided on the time points (3, 5, 7, 10, & 13 with 15 to come shortly)?
Our philosophy is to ensure manufacturers look at the performance of their implants on a regular basis. We have taken note of the history of joint replacement failures. There have been numerous examples of implants doing well for sometimes quite a while only to fall “of the cliff” often rather dramatically.
The 3 year Benchmark
When we started NICE decreed that there should be a 3A benchmark and so we certainly started with it, only to find it worked pretty well. Generally, in the UK, surgeons are reluctant to undertake revision surgery until they are absolutely sure it is necessary and there are often issues with waiting lists. This means that at say 2 years we would be concerned that a representative “early” picture would not be seen. It is realised that statistically it would be very difficult to be sure of anything with a cohort of 150 for knees and hips and 50 for shoulders but we see the 3 level benchmark as an index of cataclysmic failure. If the size of the cohort was much larger and of size that would be sensitive to statistical analysis manufacturers might not look at their data until this large number had been achieved. That could mean that a problem did not become obvious until later than would have been ideal.
Nowadays we have Beyond Compliance in the UK and so we are in a position to closely monitor most implants in their first few years in the market
The 5 year Benchmarks
There is no doubt that implants that are only just passing the 5 year benchmark are likely to struggle to make a 7 year benchmark. This is an early warning to manufacturers and they will doubtless assess their implant and the surgeons who are using it. This situation does not need to be allowed to fester until the 10 year benchmark
The 7 year benchmarks
With the experience of noting several THRs going off the rails between year 5 and 10, ODEP also introduced the 7 year benchmarks in 2003-4. The 7 year benchmark is as far as a number of implants have got before they have been removed from the market. Thus we still think there is a need for a benchmark between 5 and 10 and it also helps to maintain the momentum of the process.
The 10 year Benchmarks
When ODEP was initiated by NICE the thrust was to provide patients and surgeons with a list of implants which had 10 years of data behind them with a revision rate of <10%. NICE stipulated that any implant without a 10A rating should be part of a clinical trial.
With improvement of the reported revision rates that have occurred since ODEP started it became clear in about 2012 that the 10% revision rate was no longer acceptable and that a benchmark round about 5% was where we should be. By now many of the world’s top registries had plentiful data at 10 years. At 10 years the statisticians have much more confidence in the data
13 Year Benchmarks
With the maturity of many of the important world registries, particularly those that will cut their data at the ODEP benchmarks 13 year data is becoming increasingly available to manufacturers. Therefore, logically, it can be used to assess durability of implants in the longer term.
Obviously we have had to consider the death rate of patients with joint replacements at 13 years and in fact for the majority of implants on which we had data it was not a major problem. It may be that with implants that do not have a massive use that the manufacturer will have to wait longer to obtain the minimum number alive and unrevised at 13 years but we are making allowance for that.
Proposed 15 Year Benchmark
In the next year there will be a limited amount of 15 year registry data available from the older registries. It would seem logical to use this data for benchmarking and make available to surgeons and patients the names of the implants that are functioning at 15 years. ODEP will invite manufacturers to trial the 15 year benchmarks that we are developing in 2020. Again, consideration has been made about age etc. A recent scoping exercise proved very reassuring that 15 year benchmarking will be entirely possible
Post 15 year benchmarks
The question, where do we stop has been raised. In the next 10 years we will have 25 year data from the registries. Only nearer the time will we have any idea of what this data looks like in terms of death rate, size of cohorts etc. Doubtless the public will want to know what lasts 25 years and what doesn’t.
Benchmarking and PROMS
Now that longevity of joint replacement has for most people been established there come the question as to which prostheses actually provide the better function.
Another point is that with the lessening of follow up of TJRs being paid for by healthcare providers can we entirely rely on revision as our only metric?
Benchmarking PROMS data will be challenging. We have already looked at it for shoulders as there is no doubt that revision is a much less hard end point for failure of Total Shoulder Replacements (TSRs)
For patients with BC implants there will be Apps available from 2020. More sophisticated PROMS collecting systems are in development and are very exciting
Benchmarking for Revision Prostheses
ODEP has been evaluating the need for benchmarking revision prostheses and has scoped the issues involved. It is likely this initiative will be put to manufacturers and other interested parties in the near future.
Given the complexity and expense of revision prostheses it could be asked why ODEP has not undertaken this responsibility before. The main reason is that only more recently have registries accumulated sufficient revision data to make it remotely possible.
We have undertaken a scoping exercise to show that it is indeed possible. We have shown that “infection” does not make such an exercise impossible. ODEP for revision prostheses will probably be launched in 2020 and this date is presently under review
Benchmarking “Custom” Prostheses
There is a range of new prostheses which are in one way or another “custom” or patient matched. In theory it is not possible to benchmark these prostheses because they are all, technically, different. On the other hand unless ODEP does make it possible there will be no easy method for manufacturers to have the performance of their products performance externally validated and this they have requested. It is work in progress
We are in the process of categorizing “custom” prostheses including PSI. There is obviously several technologies / design methods being used at present from traditional custom prostheses (for tumours etc) to surface layer technology constructs (3D printing)
Spinal implants represent a completely different challenge given they are rarely removed even if they fail. The emphasis will be on PROMS
The first meeting of the ODEP 4 Spine will be convened during winter 2019-20. Beyond Compliance 4 Spine is expected to be initiated at the BOA congress
ODEP has been asked to help assess and benchmark vascular implants and cardiac implants. We have had several meetings with the clinicians and manufacturers and there is a strong feeling that the ODEP and Beyond Compliance methodology should work in these other areas
Updated November 2019