Bootstrapping Confidence Intervals in Statistics: Coverage Error Analysis - Prof. Peter G., Study notes of Data Analysis & Statistical Methods

Download Bootstrapping Confidence Intervals in Statistics: Coverage Error Analysis - Prof. Peter G. and more Study notes Data Analysis & Statistical Methods in PDF only on Docsity! METHODOLOGY AND THEORY FOR THE BOOTSTRAP (Fifth set of two lectures) Main topic of these lectures: Completion of work on confidence intervals, and sur- vey of miscellaneous topics Revision of confidence intervals Recall that η̂α is the α-level quantile of the bootstrap distribution of T ∗ = n1/2(θ̂∗− θ̂)/σ̂∗: P (T ∗ ≤ η̂α | X ) = α . A one-sided percentile-t confidence interval for an unknown parameter θ, based on the bootstrap estimator θ̂ and having nominal cov- erage α, is therefore Ĵ1 = Ĵ1(α) = (−∞, θ̂ − n−1/2 σ̂ η̂1−α) . Revision (continued) It has coverage error O(n−1): P{θ ∈ Ĵ1(α)} = α + O(n−1) . A conventional two-sided interval, for which the nominal coverage is also α, is obtained from two one-sided intervals: Ĵ2(α) = Ĵ1 { 1 2 (1 + α) } \ Ĵ1 { 1 2 (1− α) } = [ θ̂ − n−1/2 σ̂ η̂(1+α)/2 , θ̂ − n−1/2 σ̂ η̂(1−α)/2 ) . Unsurprisingly, the actual coverage of Ĵ2 also equals α + O(n−1): P{θ ∈ Ĵ2(α)} = α + O(n−1) . Coverage of two-sided percentile intervals (continued) Therefore, owing to the parity properties of polynomials in Edgeworth expansions, this two- sided percentile confidence interval has cover- age error O(n−1). The same result holds true for the “other” type of percentile confidence interval, of which the one-sided form is K̂1(α) = (−∞, θ̂ + n−1/2 σ̂ ξ̂α) . Its one- and two-sided forms have coverage P{θ ∈ K̂1(α)} = α + O(n−1/2) , P{θ ∈ K̂2(α)} = α + O(n−1) . Exercise: (1) Derive the latter property. (2) Show that, when computing percentile confidence intervals, as distinct from percen- tile-t intervals, we do not actually need the value of σ̂. (It has been included for didactic reasons, to clarify our presentation of theory, but it cancels in numerical calculations.) Discussion Therefore, the arguments in favour of percen- tile-t methods are less powerful when applied to two-sided confidence intervals. However, the asymmetry of percentile intervals will usu- ally not accurately reflect that of the statistic θ̂, and in this sense they are less appropriate. This is especially true in the case of the in- tervals K̂ (“the other percentile method”). There, when θ̂ has a markedly asymmetric distribution, the lengths of the two sides of a two-sided interval based on K̂1 will reflect the exact opposite of the tailweights. Other bootstrap confidence intervals It is possible to correct bootstrap confidence intervals for skewness without Studentising. The best-known examples of this type are the “accelerated bias corrected” intervals pro- posed by Bradley Efron, based on explicit cor- rections for skewness. It is also possible to construct bootstrap con- fidence intervals that are optimised for length, for a given level of coverage. The coverage accuracy of bootstrap confi- dence intervals can be reduced by using the iterated bootstrap to estimate coverage error, and then adjust for it. Each application gen- erally reduces coverage error by a factor of n−1/2 in the one-sided case, and n−1 in the two-sided case. Usually, however, only one application is computationally feasible. Bootstrap for time series with structural model We call the model structural because the pa- rameters describe only the structure of the way in which the disturbances drive the pro- cess. In particular, no assumptions are made about the disturbances, apart from standard moment conditions. In this sense the setting is nonparametric, rather than parametric. The best known examples of structural mod- els are those related to linear time series, for example the moving average Xj = µ + p∑ i=1 θi j−i+1 , or an autoregression such as Xj − µ = p∑ i=1 ωi (Xj−i+1 − µ) + j , where µ, θ1, . . . , θp, ω1, . . . , ωp, and perhaps also p, are parameters that have to be es- timated. Bootstrap for time series with structural model (continued, 1) In this setting the usual bootstrap approach to inference is as follows: (1) Estimate the parameters of the structural model (e.g. µ and ω1, . . . , ωp in the autore- gression example), and compute the residuals (i.e. “estimates” of the j’s), using standard methods for time series. (2) Generate the “estimated” time series, in which true parameter values are replaced by their estimates and the disturbances are re- sampled from among the estimated ones, ob- taining a bootstrapped time series X∗1, . . . , X ∗ n, for example (in the autoregressive case) X∗j − µ̂ = p∑ i=1 ω̂i (X ∗ j−i+1 − µ̂) +  ∗ j . Bootstrap for time series with structural model (continued, 1) (3) Conduct inference in the standard way, using the resample X∗1, . . . , X ∗ n thus obtained. For example, to construct a percentile-t con- fidence interval for µ in the autoregressive ex- ample, let σ̂2 be a conventional time-series estimator of the variance of n1/2µ̂, computed from the data X1, . . . , Xn; let µ̂ ∗ and (σ̂∗)2 de- note the versions of µ̂ and σ̂2 computed from the resampled data X∗1, . . . , X ∗ n; and construct the percentile-t interval based on using the bootstrap distribution of T ∗ = n1/2(µ̂∗ − µ̂)/σ̂∗ as an approximation to the distribution of T = n1/2(µ̂− µ)/σ̂ . Bootstrap for time series without struc- tural model In some cases, for example where highly non- linear filters have been applied during the pro- cess of recording data, it is not possible or not convenient to work with a structural model. There is a variety of bootstrap methods for conducting inference in this setting, based on “block” or “sampling window” methods. We shall discuss only the block bootstrap ap- proach. Block bootstrap for time series Just as in the case of a structural time series, the block bootstrap aims to construct simu- lated versions “of” the time series, which can then be used for inference in a conventional way. The method involves sampling blocks of con- secutive values of the time series, say XI+1, . . . , XI+b, where 0 ≤ I ≤ n − b is chosen in some random way; and placing them one af- ter the other, in an attempt to reproduce the series. Here, b denotes block length. Assume we can generated blocks XIj+1, . . . , XIj+b, for j ≥ 1, ad infinitum in this way. Cre- ate a new time series, X∗1, X ∗ 2, . . ., identical to: XI1+1, . . . , XI1+b, XI2+1, . . . , XI2+b, . . . The resample X∗1, . . . , X ∗ n is just the first n val- ues in this sequence. Block bootstrap for time series (contin- ued, 1) There is a range of methods for choosing the blocks. One, the “fixed block” approach, in- volves dividing the series X1, . . . , Xn up into m blocks of b consecutive data (assuming n = bm), and choosing the resampled blocks at random. In this case the Ij’s are indepen- dent and uniformly distributed on the values 1, b + 1, . . . , (m − 1)b + 1. The blocks in the fixed-block bootstrap do not overlap. Another, the “moving blocks” technique, al- lows block overlap to occur. Here, the Ij’s are independent and uniformly distributed on the values 0,1, . . . , n− b. Successes of the block bootstrap Nevertheless, the block bootstrap, and related methods, give good performance in a range of problems where no other techniques work effectively, for example inference for certain sorts of nonlinear time series. The block bootstrap also has been shown to work effectively with spatial data. There, the blocks are sometimes referred to as “tiles,” and either of the fixed-block or moving-block methods can be used. References for block bootstrap Carlstein, E. (1986). The use of subseries values for estimating the variance of a gen- eral statistic from a stationary sequence. Ann. Statist. 14, 1171–1179. Hall, P. (1985). Resampling a coverage pat- tern. Stochastic Process. Appl. 20, 231– 246. Künsch, H.-R. (1989). The jackknife and the bootstrap for general stationary observations. Ann. Statist. 17, 1217–1241. Politis, D.N., Romano, J.P., Wolf, M. (1999). Subsampling. Springer, New York. Bootstrap in non-regular cases There is a “meta theorem” which states that the standard bootstrap, which involves con- structing a resample that is of (approximately) the same size as the original sample, works (in the sense of consistently estimating the lim- iting distribution of a statistic) if and only if that statistic’s distribution is asymptotically Normal. It does not seem possible to formulate this as a general, rigorously provable result, but it nevertheless appears to be true. The result underpins our discussion of boot- strap confidence regions, which has focused on the case where the statistic is asymptoti- cally Normal. Therefore, rather than take up the issue of whether the bootstrap estimate of the statistic’s distribution is asymptotically Normal, we have addressed the problem of the size of coverage error. The m-out-of-n bootstrap (continued) The main difficulty with the m-out-of-n bootstrap is choosing the value of m. Like block length in the case of the block boot- strap, m is a smoothing parameter; large m gives low variance but high bias, and small m has the opposite effect. In most problems where we would wish to apply the m-out-of-n bootstrap, it proves to be quite sensitive to selection of m. A secondary difficulty is that the accuracy of m-out-of-n bootstrap approximations is not always good, even if m is chosen optimally. For example, when the m-out-of-n bootstrap is applied to distribution approximation prob- lems, the error is often of order m−1/2, which, since m/n → 0, is an order of magnitude worse than n−1/2. Conclusion Nevertheless, there is very substantial theoret- ical evidence that the bootstrap works quite well in a particularly wide range of statisti- cal problems, and theoretical and empirical evidence that it performs very well indeed in some settings. It is currently the only viable method for solv- ing some problems, where asymptotic approx- imations are either not available, or are poor. (Certain extreme-value problems are of this type.) For these reasons the bootstrap is a vital com- ponent of contemporary statistical methodol- ogy.