Covid-19 antibody tests face a very specific problem

With the prevalence of coronavirus infection running at about 5%, test manufacturers and regulators alike will have to guard against false positives.

Dozens of antibody tests for immunity to Covid-19 have been released onto the market around the world, so far with little regulatory oversight. This is set to change, with the FDA, CDC and NIH saying last week that they will work together to validate the accuracy of the tests on sale in the US.
But these agencies have not stated what level of accuracy they might consider sufficient, and by no means all manufacturers of these antibody blood tests have released data on their products’ sensitivity and specificity. And many of the tests for which data are available might simply not be good enough – earlier today Roche’s chief executive, Severin Schwan, said that some of the tests on the market “are not worth anything, or have very little use”.
According to data compiled by EvaluateMedTech, 29 commercial assays designed to detect antibodies to the novel coronavirus are on sale in the US, only three of which have been granted the FDA’s backing in the form of an emergency use authorisation. But none of these tests, even those with EUAs, have had their accuracy evaluated by the FDA or any other regulatory body.
Some manufacturers make their own claims for accuracy. Abbott says the IgG test it released in the US last week, when performed at least two weeks after a patient has first exhibited symptoms, has sensitivity of 100% and specificity of 99.5% (New Covid-19 test and a decent first quarter buoy Abbott, April 16, 2020). The antibody test sold in the US by Becton Dickinson, developed by BD’s partner Biomedomics, has sensitivity of 88.7% and specificity of 90.6%.
Roche intends to bring its own antibody assay out next month, so it will be interesting to see what claims that company makes for the product’s accuracy. Other companies known to be working on antibody, or serological, tests for Covid-19 include Siemens Heathineers, Danaher and Diasorin.
Major questions
There are two major questions here. Firstly, are these claims to be trusted without independent verification? Abbott and BD are reputable companies, highly unlikely to make claims they cannot justify, but many of the other antibody tests on sale around the world are from little-known groups and laboratories that might not be so scrupulous.
Secondly, how accurate do these tests need to be? Until the advent of a Covid-19 vaccine, countries aiming to lift lockdown restrictions and begin to accelerate their economic activity are going to rely heavily on antibody testing to determine which individuals can safely return to work.

Accuracy figures for selected Covid-19 antibody tests
Company	Device name	US status	EU status	Sensitivity (%)	Specificity (%)
Abbott Laboratories	Abbott Sars-CoV-2 IgG Test	Marketed	Filed	100.0	99.5
Becton Dickinson/ Biomedomics*	BioMedomics Covid-19 IgM/IgG Rapid Test	Marketed	Marketed	88.7	90.6
Cellex*	Cellex qSars-CoV-2 IgG/IgM Cassette Rapid Test	Marketed	Unclassified	93.8	95.6
Creative Diagnostics	Creative Diagnostics Sars-CoV-2 Antibody Elisa	Marketed	Marketed	94.5	100.0
CTK Biotech	OnSite Covid-19 IgG/IgM Rapid Test	Marketed	Unclassified	96.9	99.4
Epitope Diagnostics	EDI Novel Coronavirus Covid-19 IgG Elisa Kit	Marketed	Unclassified	100.0	100.0
Epitope Diagnostics	EDI Novel Coronavirus Covid-19 IgM Elisa Kit	Marketed	Unclassified	45.0	100.0
Intec Products	InTec Rapid Sars-CoV-2 Antibody (IgM/IgG)	Marketed	Marketed	95.2	98.0
Nirmidas Biotech	Nirmidas Biotech Covid-19 (Sars-CoV-2) IgM/IgG Antibody Detection Kit	Marketed	Unclassified	93.8	99.5
Ortho-Clinical Diagnostics*	Vitros Immunodiagnostic Product Anti-Sars-CoV-2 Total Reagent Pack	Marketed	Unclassified	93.7	100.0
SD Biosensor	Standard Q Covid-19 IgM/IgG Duo Test	Marketed	Unclassified	81.8	96.6
Note: All accuracy claims made by the companies. *Tests with FDA emergency use authorisation. Source: EvaluateMedTech & company websites.

The table above summarises some of the accuracy figures that selected manufacturers claim for their serological Covid-19 tests. But the validation tests these companies have performed varied widely in size; Abbott’s antibody assay was tested on 1,200 specimens, whereas Epitope’s tests were run on 54 samples from healthy people and just 20 and 30 cases of PCR-confirmed Covid-19 for the IgM and IgG tests respectively. The usual concerns about comparing data from different studies are vastly magnified here.
And accuracy needs to be high. The prevalence of Covid-19 is estimated at around 5% in the US, and at this low a level the risk of false positives becomes a major problem. If a serological test has 90% specificity, its positive predictive value will be 32.1% – meaning nearly 70% of positive results will be false. At this same disease prevalence, a test with 95% specificity will lead to a 50% false positive rate. Only at 99% specificity does the false positive rate become anywhere near acceptable, and even here 16% of positive results would still be wrong.

Predictive value of a theoretical test at disease prevalence of 5%
Sensitivity and specificity of 90%	NPV	99.4%	False negative rate	0.6%
	PPV	32.1%	False positive rate	67.9%
Sensitivity and specificity of 95%	NPV	99.7%	False negative rate	0.3%
	PPV	50.0%	False positive rate	50.0%
Sensitivity and specificity of 99%	NPV	99.9%	False negative rate	0.1%
	PPV	83.9%	False positive rate	16.1%
NPV & PPV = negative & positive predictive values. Source: Evaluate Vantage calculations.

The FDA will need to set demanding accuracy standards for these products if it is to avoid large numbers of people with no immunity to the virus being sent back into contact with others. The agency has said it will take appropriate action against companies making or distributing unvalidated tests or those making false claims, such as issuing warning letters requesting that companies stop their unlawful promotion.
If a company can demonstrate decent, reproducible levels of accuracy, there is money to be made. Leerink analysts assume a price of $5-10 for strip-based assays similar to pregnancy tests – the kind being sold by Cellex and Chembio, for example – and $25-50 for high-throughput testing done at a central lab, such as that offered by Abbott and, next month, by Roche. If half the US population is tested, the market in that country will exceed $3bn.
Companies used to working at scale will obviously be best placed to gain market share. Abbott says it will be able to conduct 20 million tests per month by June, and Roche said that it will manage five times that.
There is one last headache for those who believe that serological testing is the means through which normality will resume. As Bernstein analysts point out, even a perfect test would still only allow the 5% of the population that had been infected back to work – hardly a respectable basis for kickstarting an economy.
https://www.evaluate.com/vantage/articles/analysis/spotlight/covid-19-antibody-tests-face-very-specific-problem