Resampling Approaches and Confidence Intervals in Regression and Correlation Models - Prof, Study notes of Data Analysis & Statistical Methods

Download Resampling Approaches and Confidence Intervals in Regression and Correlation Models - Prof and more Study notes Data Analysis & Statistical Methods in PDF only on Docsity! METHODOLOGY AND THEORY FOR THE BOOTSTRAP (Sixth set of two lectures) Main topic of these lectures: Bootstrap methods in linear regression Regression model Assume we observe pairs (x1, Y1), . . . , (xn, Yn), generated by the model Yi = g(xi) + ǫi , (1) where g is a function that might be determined either parametrically or nonparametrically, and the errors ǫi have zero mean. 1 Regression model (cont.) In the study of regression we take the explana- tory variables xi to be fixed, either because they are pre-determined (e.g. were regularly spaced) or are conditioned upon. In this case, the only source of randomness in the model is the errors, ǫi, and so it is those that we resample, in form of residuals, when implementing the bootstrap. Our choice of lower-case notation for the explanatory vari- ables reflects this view. 2 General definition of slope Although these exceptional coverage proper- ties apply only to estimates of slope, not to estimates of intercept parameters or means, slope may be interpreted very generally. For example, in the polynomial regression model Yi = c + xi d1 + . . . + x m i dm + ǫi , where we observe (xi, Yi) for 1 ≤ i ≤ n, we re- gard each dj as a slope parameter. A one-sided percentile-t interval for dj has coverage error O(n−3/2), although a one-sided percentile-t in- terval for c or for E(Y |x = x0) = c + x0 d1 + . . . + x m 0 dm has coverage error of size n−1. 5 Why is slope favoured especially? The reason for good performance in the case of slope parameters is the extra symmetry con- ferred by design points. Note that, in the polynomial regression case, we may write the model equivalently as Yi = c ′ + (xi − ξ1) d1 + . . . + (x m i − ξm) dm + ǫi , where ξj = n −1 ∑ i x j i and c ′ = c + ξ1 d1 + . . . + ξm dm. The extra symmetry arises from the fact that n∑ i=1 (x j i − ξj) = 0 for 1 ≤ j ≤ m. 6 Correlation model The results implying good coverage accuracy hold under the regression model, but not nec- essarily for the correlation model. For example, in the case of the linear correlation model, the symmetry discussed above will persist provided that E    n∑ i=1 (Xi − ξ1) ǫ k i    = 0 (1) for sufficiently large k. Now, ǫi = Yi − g(Xi), and as a result, (1) will generally not hold for k ≥ 1. This means that, under the cor- relation model, the conventional properties of bootstrap confidence intervals hold; the special properties noted for regression, when estimat- ing slope parameters, are not valid. However, if the errors ǫi are independent of the explanatory variables Xi then (1) will hold for each k, and in such cases the enhanced fea- tures of the regression problem persist under the correlation model. 7 Estimating quantiles of distribution of d̂ Let ĉ∗, d̂∗ and σ̂∗ have the same formulae as ĉ, d̂ and σ̂, respectively, except that we replace Yi by Y ∗ i throughout. The bootstrap versions of S = n1/2 (d̂ − d)σx/σ , T = n1/2 (d̂ − d)σx/σ̂ are S∗ = n1/2 (d̂∗ − d̂)σx/σ̂ , T ∗ = n1/2 (d̂∗ − d̂)σx/σ̂ ∗ , respectively. We estimate the quantiles ξα and ηα of the dis- tributions of S and T by ξ̂α and η̂α, respectively, where P(S∗ ≤ ξ̂α | X) = α , P(T ∗ ≤ η̂α | X) = α , and X = {(x1, Y1), . . . , (xn, Yn)} denotes the dataset. 10 Bootstrap confidence intervals for d One-sided bootstrap confidence intervals for d, with nominal coverage α, are given by Î11 = ( −∞, d̂ − n−1/2 (σ/σx) ξ̂1−α ) , Î12 = ( −∞, d̂ − n−1/2 (σ̂/σx) ξ̂1−α ) , Ĵ1 = ( −∞, d̂ − n−1/2 (σ̂/σx) η̂1−α ) . These are direct analogues of the intervals Î11, Î12 and Ĵ1 introduced earlier in non-regression problems. In particular, Î12 and Ĵ1 are standard percentile-method and percentile-t bootstrap confidence regions. Following the line of argument given earlier, we would expect Î11 and Ĵ1 to have coverage error O(n−1). In fact, they both have coverage error equal to O(n−3/2). However, Î11 is not of practical use, since it depends on the unknown σ, so we shall not treat it any further. Likewise, we would expect Î12 to have cov- erage error of size n−1/2. However, we shall show that the error is actually of order n−1. 11 Bootstrap confidence intervals for d (cont.) Note that, although Î12 involves the variance estimator σ̂, it can be constructed numerically without resorting to computing σ̂. Indeed, Î12 is identical to the interval Î12 = ( −∞, d̂ − ŵ1−α ) , where ŵ1−α is the standard percentile-method estimator of w1−α, the latter defined by P(d̂ − d ≤ w1−α) = 1 − α . In particular, ŵ1−α is defined by P(d̂∗ − d̂ ≤ ŵ1−α | X) = 1 − α . The interval Ĵ1 is a standard percentile-t boot- strap confidence interval. 12 Why does P1 equal Q1? (cont. 1) Making use of the approximation T = S (1 − ∆) + Op(n −1) , (1) the symmetry property n∑ i=1 (xi − x̄) = 0 , (2) and the representation S = n−1/2 σ−1x σ −1 n∑ i=1 (xi − x̄) ǫi , it is readily proved that E{S (1 − ∆)}j = E(Sj) + O(n−1) (3) for j = 1,2,3. Exercise: Derive (3) for j = 1,2,3. 15 Why does P1 equal Q1? (cont. 2) Therefore, the first three cumulants of S and S (1 − ∆) agree up to and including terms of order n−1/2. It follows that Edgeworth expansions of the distributions of S and S (1 − ∆) differ only in terms of order n−1. In view of (1), the same is true of the distributions of S and T : P(S ≤ u) = P(T ≤ w) + O(n−1) . Therefore, the n−1/2 terms in the expansions must be identical; that is, P1 = Q1. The chief ingredient in this argument is the symmetry property (2). It, and its analogues, guarantee that, in the problem of slope estima- tion for general regression problems, P1 = Q1. 16 Consequences of the property P1 = Q1 The identity P1 = Q1 implies that, to first or- der (i.e. up to and including terms of order n−1/2), estimating the distribution of S is the same as estimating the distribution of T . As we saw earlier in non-regression problems, the percentile method estimates the distribu- tion of S, whereas the percentile-t method es- timates the distribution of T . The fact that, in the setting of estimating slope in regression, P1 = Q1, means that these two techniques give the same results up to and including terms of order n−1/2. They differ only in terms of order n−1, and terms of higher order. Therefore, since one-sided confidence intervals based on the percentile-t method have cover- age error equal to O(n−1), the same must be true for confidence intervals based on the per- centile method. 17 Derivation of (4) (cont. 1) The bootstrap version of the Taylor expansion is P(T ∗ ≤ u | X) = Φ(u) + n−1/2 Q̂1(u)φ(u) +n−1 Q̂2(u)φ(u) + . . . , where Q̂1(u) = 1 6 γ̂ γx (1 − u 2) and γ̂ = E(ǫ̂/σ̂)3. Now, the solutions ηα and η̂α, of the respective equations P(T ≤ ηα) = α , P(T ∗ ≤ η̂α | X) = α , admit the Cornish-Fisher expansions ηα = zα + n −1/2 Qcf1 (zα) + n −1 Qcf2 (zα) + . . . , η̂α = zα + n −1/2 Q̂cf1 (zα) + n −1 Q̂cf2 (zα) + . . . . 20 Derivation of (4) (cont. 2) Cornish-Fisher expansions: ηα = zα + n −1/2 Qcf1 (zα) + n −1 Qcf2 (zα) + . . . , η̂α = zα + n −1/2 Q̂cf1 (zα) + n −1 Q̂cf2 (zα) + . . . . On subtracting these expansions we deduce that η̂α − ηα = n −1/2 {Q̂cf1 (zα) − Q cf 1 (zα)} +n−1 {Q̂cf2 (zα) − Q cf 2 (zα)} + . . . = n−1/2 {Q1(zα) − Q̂1(zα)} +Op(n −3/2) , where we have used the fact that Qcf1 = −Q1, Q̂cf1 = −Q̂1 and Q̂ cf 2 = Q cf 2 + Op(n −1/2). 21 Derivation of (4) (cont. 3) From previous pages: Q̂1(u) − Q1(u) = 1 6 (γ̂ − γ) γx (1 − u 2) , (5) η̂α−ηα = n −1/2 {Q1(zα)−Q̂1(zα)}+ Op(n −3/2) . (6) It may be proved by Taylor expansion that γ̂ = n−1 ∑ i {ǫi − ǭ − (xi − x̄) (d̂ − d)} 3 [n−1 ∑ i {ǫi − ǭ − (xi − x̄) (d̂ − d)} 2]3/2 = γ + n−1/2 U + Op(n −1) , (7) where U = n−1/2 n∑ i=1 { (δ3i − γ) − 3 2 γ (δ 2 i − 1) − 3 δi } and δi = ǫi/σ. Combining results (5)–(7) we deduce that η̂α − ηα = −n −1 1 6 U γx (1 − z 2 α) + Op(n −3/2) . 22 The other percentile-method interval Recall that the percentile-method interval Î12 is based on bootstrapping d̂ − d; that is, it is based on approximating the distribution of this quantity by the conditional distribution of d̂∗− d̂. The “other” percentile method is based on us- ing the conditional distribution of d̂∗ to approx- imate the distribution of d̂. It leads to the in- terval K̂1 = ( −∞, d̂ + n−1/2 (σ̂/σx) η̂α ) = (−∞, ζ̂α) , where ζ̂α is an approximation to ζα, these two quantities being defined by P(d̂∗ ≤ ζ̂α | X) = α , P(d̂ ≤ ζα) = α . However, K̂1 has coverage error of size n −1/2, not n−1. In particularly, K̂1 does not enjoy the accuracy of the percentile-method interval Î12. This is a consequence of it addressing the wrong tail of the distribution of d̂. 25 Properties of confidence intervals for the conditional mean, y0 = E(Y |x = x0), and the intercept, c Recall that y0 = c + x0 d, which in turn equals c when x0 = 0. Therefore we can treat confi- dence intervals for c as a special case of those for y0. Recalling that ŷ0 = ĉ + x0 d̂, redefine S = n1/2 (ŷ0 − y0)/(σ σy) , T = n1/2 (ŷ0 − y0)/(σ̂ σy) ; and taking ŷ∗0 = ĉ ∗ + x0 d̂ ∗, redefine S∗ = n1/2 (ŷ∗0 − ŷ0)/(σ̂ σy) , T ∗ = n1/2 (ŷ∗0 − ŷ0)/(σ̂ ∗ σy) . 26 Properties of confidence intervals for y0 and c (cont.) Percentile-method and percentile-t confidence intervals for y0 and c are given respectively by Î12 = ( −∞, ŷ0 − n −1/2 (σ̂/σy) ξ̂1−α ) , Ĵ1 = ( −∞, ŷ0 − n −1/2 (σ̂/σy) η̂1−α ) , where σ2y = 1 + σ −2 x (x0 − x̄) 2 and we define ξ̂1−α and η̂1−η by P(S∗ ≤ ξ̂α | X) = α , P(T ∗ ≤ η̂α | X) = α , for the new versions of S∗ and T ∗. The intervals Î12 and Ĵ1 have coverage errors O(n−1/2) and O(n−1), respectively. These re- sults, unlike those for slope, are conventional. 27