Study: FDA Has ‘No Tradition or Structure to Ensure Consistency’ in Decisions

How should the FDA approach drug approval decisions when the evidence for a product’s efficacy isn’t overwhelming? One place to start would be a rigorous examination of past decisions to bring some consistency and predictability to the process — which the agency currently doesn’t do and couldn’t if it wanted to, researchers said in a blistering critique of the agency’s processes.

From 2013 to 2018, a total of 22 drugs were initially rejected because the FDA considered the efficacy evidence inadequate but were eventually approved upon resubmission. For seven of these, “the approval did not require new evidence but rather new interpretations of the original evidence,” according to Perrine Janiaud, PhD, of Stanford University in California, and colleagues, writing in Annals of Internal Medicine.

The study — funded indirectly by the FDA, with two co-authors who were agency employees when it began — also found that eight drugs were approved even though problems leading to the initial denials were not all resolved.

Most significantly, “[n]o FDA decisions cited reasoning used in previous decisions.”

Although the study focused on a 6-year period ending in 2018, Janiaud and colleagues saved some of their harshest critiques for the decision making, or at least the record of it, regarding the Alzheimer’s disease drug aducanumab (Aduhelm) approved in June of this year.

“The aducanumab decision was shocking to many because of its apparent break with FDA precedent,” Janiaud and colleagues wrote. “It seemed to apply a lower evidential standard than previous decisions in accepting weak, post hoc statistical evidence and an unproven surrogate end point (amyloid reduction). This was manifest in the unanimous [advisory committee] vote against approval, the strong internal dissent from the biostatistics office, the controversy in the scientific media, and the subsequent resignation of [three advisory committee] members.”

While a top official argued that the decision was based on a “standard we use fairly frequently,” a 68-page FDA memorandum summarizing the rationale cited no prior decisions, the researchers pointed out.

“A perception of inconsistency with prior decisions breeds distrust in FDA decisions and belief that they are not being driven primarily by evidence, threatening the FDA’s scientific, public, and political standing,” Janiaud’s group argued.

Piling on was one of those advisory committee members who quit in protest. In an accompanying editorial, Joel Perlmutter, MD, of Washington University in St. Louis, suggested that industry influence was too strong at the FDA, “with more than half of the FDA budget provided by applicant companies.”

Not only should the “financial tether between the FDA and industry” be severed, but the agency needs “prospective rules for transparency,” as well as more funding from Congress, Perlmutter wrote.

For the formal analysis of approval decisions in 2013-2018, Janiaud and colleagues pulled FDA documentation on 912 applications that were reviewed during this period. Of those, 117 were subject to multiple review cycles, with 22 scrutinized specifically “because of issues related to clinical efficacy,” the group explained.

A variety of issues led to the initial rejections. Janiaud and colleagues identified 12 different categories, such as using inconsistent endpoints within an application; studying different doses; running studies with too few patients; poor statistical analysis; and claiming non-inferiority based on excessively wide margins.

For 15 of the 22 with efficacy problems, the manufacturers supplied additional data, sometimes from all-new pivotal studies. But that was not the case for the other seven.

In eight cases, products were eventually approved without documenting resolution of all the problems identified in the initial rejections:

• Flibanserin (Addyi)
• Inhaled insulin (Afrezza)
• Daclatasvir (Daklinza)
• Macimorelin (Macrilen)
• Droxidopa capsules (Northera)
• Buprenorphine (Probuphine)
• Methylene blue (ProvayBlue)
• Coagulation factor Xa (Andexxa)

These applications had deficiencies in as many as seven separate categories initially.

Janiaud and colleagues analyzed one of these in detail: droxidopa capsules, for which the manufacturer, Chelsea Therapeutics, submitted its first application in 2011, which was finally approved in 2014. FDA officials identified various problems, including that the final endpoint in the initial pivotal study was not the one originally agreed to, that the magnitude of effect on the chosen endpoint was of only “marginal clinical significance,” and that even that effect wasn’t shown to last more than 1 week.

Chelsea did submit an additional pivotal study, but it, too, shifted the primary endpoint midstream, and FDA staff determined that the treatment benefit was marginal. But an advisory committee favored approval largely on the basis of “patient testimonies,” according to Janiaud’s group, and the relevant office director at FDA approved the drug because the application came through the accelerated approval pathway, under which a product can be OK’d if the data indicate clinical benefit is “reasonably likely.”

That standard — “reasonably likely” — was also cited in the aducanumab decision. In that case, Janiaud and colleagues said, the FDA could have made the connection to the droxidopa approval as a relevant precedent. That it did not do so is indicative of the agency’s larger problem: that “each decision [in the cases analyzed] invoked a bespoke rationale,” often lacking even the feeblest gestures toward showing consistent standards.

“No decisional memos referred to the reasoning used in other approvals, meaning that the FDA does not have a tradition or structure to ensure consistency in its decisions,” the group wrote.

And the reason for this, they continued, is that the agency has no way to identify potential precedents even if it wanted. “The FDA’s electronic database is largely unstructured, taking hours or days to read through case documentation. It is very difficult to find related cases, which requires that evidential characteristics and uncertainties be abstracted and codified, as in the tables and figures” Janiaud and colleagues provided in their Annals paper.

Yet another problem, the group argued, is what they called the FDA’s “balkanization,” with separate centers and divisions within centers that to a great extent operate as independent fiefs. In several of the 22 cases analyzed in the study, drug sponsors resubmitted their applications to a different division. “Multiple FDA staff leaders and collaborators confirmed that the FDA has weak structures to support institutional memory,” the authors wrote.

This lack of concern for precedent, Janiaud and colleagues noted, might actually have helped aducanumab gain approval: officials did not need to consider that their ad-hoc rationale would be influential in future product reviews.

Janiaud and colleagues did acknowledge a significant limitation to their analysis, which was that it relied on the drug approval packages available from the FDA — which do not include verbatim communications between the agency and applicants, nor full details of the agency’s internal deliberations.