Fifty shades of QE: Robust evidence

Fabo, Jancokova, Kempf, and Pastor (2021) show that papers written by central bank researchers find quantitative easing (QE) to be more effective than papers written by academics. Weale and Wieladek (2022) show that a subset of these results lose statistical significance when OLS regressions are replaced by regressions that downweight outliers. We examine those outliers and find no reason to downweight them. Most of them represent estimates from influential central bank papers published in respectable academic journals and are frequently cited. Moreover, we show that these papers have supported policy communication by the world’s leading central banks and shaped the public perception of the effectiveness of QE. New evidence from quantile regressions further supports the results in Fabo et al. (2021).

The effectiveness of quantitative easing (“QE”) has been a subject of intense debate in both academic and policy circles. A significant part of QE research originates in central banks (Martin and Milas, 2012). Central banks evaluating their own policies is not unlike pharmaceutical firms evaluating their own drugs. Both have skin in the game.

Fabo et al. (2021; henceforth “FJKP”) compare the findings of central bank researchers and academics regarding the effectiveness of unconventional monetary policy. They construct a dataset comprising 54 studies that analyze the effects of unconventional monetary policy (“QE” for short) on output or inflation in the U.S., UK, and the euro area. Figure 1 below shows the histograms of the estimated effects on output for the subsample of papers with no central bank authors (“Not CB”) versus papers with at least one central bank author (“CB”), reported by FJKP. The figure shows that papers by central bankers report systematically larger effects of QE on output, as indicated by a larger right tail of the distribution of central bank papers. FJKP find a similar pattern for inflation. Central bank papers are also more likely to report QE effects on output that are significant, both statistically and economically, and they use more favorable language in their abstracts. In sum, papers written by central bankers find QE to be more effective than papers written by academics. Moreover, central bank researchers who report larger QE effects on output experience more favorable career outcomes.

Figure 1: The Effect of QE on Output by Central Bank Affiliation.

Source: Fabo, Jancokova, Kempf, and Pastor (2021).

Weale and Wieladek (2022) reexamine FJKP’s analysis. First, they reject the null hypothesis that the residuals in FJKP’s OLS regressions are normally distributed. Second, Weale and Wieladek (2022) rerun parts of FJKP’s analysis using two “robust regression” methods, the median regression and the MM/MS regression, which downweight the influence of large residuals compared to OLS. Based on these methods, Weale and Wieladek (2022) obtain the same conclusions as FJKP regarding language sentiment but they cannot reject the null hypothesis that central bankers and academics report the same quantitative effects of QE on output and inflation. The finding that the median regressions in Weale and Wieladek (2022) produce smaller point estimate compared to OLS is consistent with Figure 1 from FJKP, which shows that the most striking differences in the distributions of central bank and academic studies is in the right tail. Weale and Wieladek (2022) do not use robust regressions to reexamine FJKP’s evidence on significance reporting or career progression.

We are grateful to Weale and Wieladek (2022) for their illuminating analysis. It is clearly important to examine the robustness of the empirical findings reported in the economics literature. Yet such studies are few and far between, perhaps due to insufficient incentives for the authors. By providing such analysis, Weale and Wieladek (2022) have performed valuable service to the profession.

In our new paper (Fabo et al., 2023), we build on Weale and Wieladek (2022) to shed more light on the robustness of FJKP’s results. As we explain in more detail below, we disagree with Weale and Wieladek (2022) on the relevance of normally distributed residuals for FJKP’s analysis, as well as on the need to downweight outliers. We show that several papers with the largest reported effects on output and inflation have been very influential in the academic literature, and they have supported policy communication by the world’s leading central banks and shaped the public perception of the effectiveness of QE. We also present new evidence from quantile regressions, assessing the effect of central bank affiliation not only at the median but across the full conditional distribution.

Normality

The OLS estimation method, used by FJKP, does not assume normally distributed residuals. The normality of the residuals is neither necessary nor sufficient for OLS estimates to be consistent. In other words, there is nothing inherently wrong with using OLS estimates when the residuals are not normally distributed.

Normality is not needed to conduct inference about OLS estimates, either. Given their relatively small sample, FJKP do not rely on asymptotic standard errors; instead, they use a wild cluster bootstrap. Bootstrapping methods are based on the empirical distribution of the residuals. Therefore, Weale and Wieladek (2022)’s rejection of the normality of the residuals in some of FJKP’s OLS regressions does not invalidate the OLS estimates or the inference about them.

Outliers and OLS

Nonetheless, Weale and Wieladek (2022)’s evidence of non-normality is useful because it suggests that some of the datapoints could potentially be interpreted as outliers. A well-known feature of OLS regressions is that its estimates can be sensitive to outlying residuals. The reason is that the quadratic loss function inherent to OLS increases sharply with the magnitude of the residuals. To moderate this sensitivity, robust regressions use different loss functions that are less responsive to outliers. For example, the median regression, which is used by Weale and Wieladek (2022), minimizes the sum of absolute residuals rather than their squares.

However, it is not clear that downweighting the outliers compared to OLS is the right thing to do. After all, OLS regressions are used far more frequently than robust regressions. Whether one wants to downweight the outlying observations should depend on one’s belief regarding the validity of these observations. If one believes that the outliers are credible observations (as opposed to, say, data errors) coming from the same data-generating process as the remaining observations, then there is no need to downweight them.

In fact, downweighting the outliers can be the wrong thing to do. For example, a scientist analyzing seismic activity would be ill-advised to downweight major earthquakes, because these outliers are the most important observations. Similarly, outliers are likely to be of particular interest to a doctor analyzing a patient’s heart rate history. In our context, papers reporting large effects of QE can be very important in forming the perception of the effectiveness of QE among academics, policy makers, and the general public.

The Impact of Outliers in FJKP

We find that the outliers in FJKP’s regressions have been highly influential on several dimensions. All of them come from studies that appear to be competent, written by credible authors. Many of these studies have been published in respectable academic journals, such as the American Economic Journal: Macroeconomics, Economic Policy, Journal of Monetary Economics, and Journal of Money, Credit and Banking. Moreover, several of them have received hundreds of citations. For example, two of the five papers that report the largest peak effect of QE on output have been cited 741 and 316 times, respectively, as of 30 December 2022.

The outliers in FJKP may have also been highly influential outside of the academic literature. Public speeches by prominent central bankers offer a rare glimpse into the set of papers that influence policy analysis inside a central bank. We therefore search the speeches of prominent central bankers, available on the websites of the Bank of England, the ECB, and the Federal Reserve, for references to the studies that report the largest effects on output and inflation.

We find that many of the outliers have been cited by prominent central bankers. In one of his speeches, former Chair of the Federal Reserve Ben Bernanke bases his statement about securities purchase programs having provided “significant help for the economy” on the findings of one of the outlier studies in FJKP (Bernanke (2012)). Huw Pill, member of the Bank of England’s Monetary Policy Committee, cites one of the outlier papers as having influenced policy analysis at the Bank of England (Pill (2022)). We find other examples of outlier studies being mentioned in speeches by the Fed’s Janet Yellen, the ECB’s Benoît Cœuré and Isabel Schnabel, and the Bank of England’s Andy Haldane.

Papers reporting large effects of QE can also help form the public perception of the effectiveness of QE, via their media coverage. Two studies that frequently appear in our list of papers reporting the largest estimated effects of QE are prominently cited in the Financial Times (2015): “The good news is that, by most accounts, QE appears to have succeeded at boosting growth and lifting inflation. Martin Weale, a member of the BoE’s interest-rate setting Monetary Policy Committee, found asset purchases worth 1% of national income boosted UK gross domestic product by about 0.18% and inflation by 0.3%. A study by John Williams, president of the San Francisco Federal Reserve, concluded that asset purchases had reduced the US unemployment rate by 1.5 percentage points by late 2012 and helped the economy avoid deflation.”

Another article in the Financial Times (2017) suggests that the findings from one of the outlier studies, which was also cited prominently by Janet Yellen, were “near the consensus of Fed thinking on the subject” at the time:

“This study, by Eric Engen, Thomas Laubach and David Reifschneider, presents conclusions that may be near the consensus of Fed thinking on the subject at present. The Fed study suggests that the effect of the entire QE programme was to reduce 10 year term premium, and therefore the bond yield, by 120 basis points in 2013. This is estimated to have reduced US unemployment by about 1.25 percentage points and increased inflation by about 0.5 percentage points.” We provide several other examples of outlier studies being cited by the Financial Times and the Wall Street Journal.

Overall, it is far from clear to us that downweighting these highly influential papers, which is what robust regressions do, is appropriate. In fact, one might argue that such studies should be overweighted, given their disproportionate influence.

New Evidence from Quantile Regressions

We also extend Weale and Wieladek (2022)’s analysis by estimating quantile regressions. Unlike Weale and Wieladek (2022), we not only assess the effect of central-bank affiliation at the median, but across a wide range of quantiles.

Figure 2 below reports the results from our quantile regressions for the estimated effects on output. Across all four panels, we find a positive coefficient on central-bank affiliation (CB Affiliation) at every quantile. Moreover, the effect of CB Affiliation is substantially larger above the 80th percentile. For example, for the total peak effect on output (Panel A), the effect of CB affiliation is equal to 1.87 percentage points at the 90th percentile. As a comparison, it is equal to 0.77 percentage points at the 10th percentile.

Figure 2: Effects of QE on Output: Estimates from Quantile Regressions

Source: Fabo, Jancokova, Kempf, and Pastor (2023).

For the estimated effect on inflation, reported in Figure 3 below, we find a similar pattern: the effect of CB Affiliation is positive at all quantiles, and it tends to be larger for higher quantiles, especially for the peak effect (Panels A and C).

Figure 3: Effects of QE on Inflation: Estimates from Quantile Regressions


Source: Fabo, Jancokova, Kempf, and Pastor (2023).

Overall, our quantile regression analysis shows that the results from FJKP are qualitatively robust. They also add a more comprehensive view to the median regressions estimated by Weale and Wieladek (2022), by highlighting the important variation in the right tail of the conditional distribution.

Conclusion

We examine the outliers in FJKP’s data and find no reason to downweigh them. The outlying estimates of the effects of QE come from credible papers, most of which have been published in respectable academic journals and highly cited by researchers. They have also been cited in leading media outlets and in public speeches of prominent central bank policy makers. We do not find it desirable to put less weight on estimates from influential papers that have impacted the perception of the effectiveness of QE in the eyes of top central bank officials and the general public.

We extend Weale and Wieladek (2022)’s analysis by re-estimating FJKP’s regression specifications using quantile regressions, for a wide range of quantiles. Just like FJKP, we estimate a positive difference between the QE effects estimated by central bankers and academics. While the point estimate of this difference is mostly statistically insignificant, it is consistently positive across quantiles. FJKP’s findings thus emerge not only from OLS regressions but also from quantile regressions. The point estimate tends to be larger, and more often significant, at the top quantiles. Further research is needed to understand this variation in the point estimates across the conditional distribution.