PowerPoint Presentation

Transcription

1 13. Linear Regression and Correlation 數 1

2 Outline Data: two continuous measurements on each subject Goal: study the relationship between the two variables PART I : correlation analysis Study the relationship between two continuous variables. Steps : Scatter diagram Correlation coefficient : Calculation, meaning, hypothesis testing PART II : linear regression Construct a linear equation between variables. Model building Model estimating : Confidence intervals and prediction intervals Model fitting: Strength of the linear association,coefficient of determination

3 Recall : In Ch.11 and 1, X : nominal variable. Ex. Gender(binary), brand(3-level) Y : response variable(cont. or binary). Ex. Score, success-failure, yield, Q : whether X and Y are correlated? A : If Y is continuous, comparing the population means of Y in the groups divided by X. Ex : 藍 綠 --Z-test, T-test, ANOVA F-test 3

4 Recall : In Ch.11 and 1, Q : whether X and Y are correlated? A : If X and Y are binary, compare the population proportions of Y in the two groups divided by X. Ex : 參 行 例 女 參 行 例 When sample sizes are large, Z-test is used. Q : How to determine the correlation if X and Y are both continuous? -- correlation and regression analysis! 4

5 Data : A sample of n sets of observation. There are k continuous variables measured in each observation. Example. Surveyed n=10 students, k=3 scores are recorded. Questions : any association between scores? 5

6 What is correlation analysis? Study the relationship between several continuous variables. Measure the strength of the association between variables. Correlation analysis consists : Step 1. Scatter diagram : Plot (X1, X) Step. Coefficient of correlation : 6

7 EXCEL Conclusion : 1. 不 率 0 行. 3. 7

8 Population coefficient of correlation, : A measure of the strength of the linear relationship between two variables. Definition: population correlation coefficient (x µ µ ρ = σ σ Estimation : sample correlation coefficient Sxy r = SS x y x )(y y)p(x, y) x y where S xy n (xi x)(yi y) xiyi nxy i= 1 i= 1 = = n 1 n 1 n n (xi x) (yi y) i= 1 i= 1 y S x =,S = n 1 n 1 n 8

9 Properties : -1 r 1 Positive linear association : r > 0 Negative linear association : r < 0 no linear relation : r 0 (! Other relation may exist) Strongly positive linear association : r 1 Strongly negative linear association : r -1 9

10 10

11 Why such definition? If there is a strongly positive linear association, when x is large, y is large, then we have a large positive value of Sxy. If there is a strongly negative linear association, when x is large, y is small, then we have a large negative value of Sxy. If there is no relation, when x is large, some y are large, some y are small, then Sxy 0, r 0. 11

12 EXCEL 欄 欄 欄 數 數 數 數 數 數 異數 異數 異數 度 度 度 數 數 數 1

13 EXCEL Conclusion : 1. 數 = 數 = 數 =

14 Recall in a single population with one variable : Population : N= subjects X=measurement =population mean H0 : = 0? Unknown! Random sampling Sample : n subjects X=observations calculation X =sample mean X 0 t/z= SE(X) -- a t/z-test!. 14

15 Now we have a single population with two variables : Population : N= subjects Measurements:(X, Y) =population coef. of correlation H0 : = 0? Unknown! Random sampling calculation r = sample coef. of correlation Sample : n subjects Observations: (X,Y) t r 0 = SE(r) -- a t-test! 15

16 Testing the significance of the correlation coefficient Testing the null hypothesis of no correlation : =0 Step 1. State the hypotheses H0 : no correlation v.s. H1: correlated H0 : = 0 v.s. H1 : 0 Step. Select the significance level 16

17 Step 3. Determine the test statistic r 0 r r n t = = = SE(r) (1 r ) /(n ) 1 r Note that under null hypothesis, t ~ t-distribution with d.f.=(n-) Step 4. Formulate the decision rule A two-sided test; A t-test; With significance level, H0 should be rejected if t > t /,n- or t <- t /,n- Step 5. Collect data, compute t-value, draw conclusion 17

18 Example. At =0.05, n=10, df=10-=8, t (0.05,8) =.306 Test 1 : ( v.s. ) H0 : 1= 0 v.s. H1 : 1 0 Since r1=0.033, n=10, r t = n = = 1 r1 1 (0.033) Since.306< t=0.093 <.306, H0 is not rejected. Conclusion : there is no sufficient evidence to reject the null hypothesis of no correlation. 18

19 Example. At =0.05, n=10, df=10-=8, t (0.05,8) =.306 Test : ( v.s. ) H0 : = 0 v.s. H1 : 0 Since r=0.7151, n=10, t = r n = = 1 r 1 (0.7151).89 Since t=.89>.306, H0 is rejected. Conclusion : there is sufficient evidence to reject the null hypothesis of no correlation. 19

20 Example. At =0.05, n=10, df=10-=8, t (0.05,8) =.306 Test 3: ( v.s. ) H0 : 3= 0 v.s. H1 : 3 0 Since r3=0.3755, n=10, r t = n = = 1 r3 1 (0.3755) 1.15 Since.306< t=1.15 <.306, H0 is not rejected. Conclusion : there is no sufficient evidence to reject the null hypothesis of no correlation. 0

21 ! A word of caution : for H0 of no correlation been rejected,! Only linear relationship between variables are ascertained.! Quadratic? Cubic?! No cause and effect ( ) is established.!!!?!! Spurious( ) correlations :! 來 數! 來 數量! 數 年 1

22 Variables : PART II. Linear Regression Analysis X=Independent variable(s), explanatory variable, predictor, 數, 數 Y=Dependent variable, response variable, 數, 數 To be predicted or estimated. Regression analysis : Develop an equation/function that allows us to estimate/predict Y based on X. Example. X= Y 若 60 數

23 ANOVA Regression Recall : In a one-way ANOVA AGE vs. INCOME The whole population are classified into three subpopulations by AGE A young-population. A middle-age-population. A senior-population The INCOMEs of all sub-populations are Normally distributed with same variance Research question: The mean INCOMEs, income,are the same? 3

24 ANOVA Regression Recall : In a simple linear regression model, (X) vs. (Y) The whole population are classified into many sub-populations by (X) X=0-population; X=1-population;..., X=100-population. The (Y) of all sub-populations are Normally distributed with same variance Research question: The mean s, Y, are the same? Establish the relationship between Y and X 4

25 Regression Model: (P449) 1. Given each value of X, there is a group of Ys. X X=60 Y= X=50 Y=. These Y values ~ normal distribution. At X=60, Y~ N( µ Y X = 60, σ ) At X=50, Y~ N(, σ ) µ Y X = 50 5

26 3. The means of these normal distributions is a linear function of x 數 X 數 數 X 數 µ Y x = α + βx Example ( ) Y = x 6

27 4. The standard deviations of these normal distributions are all the same. (independent with x) 異數 At X=60, Y~ At X=50, Y~ N( µ Y X = 60, σ ) N(, σ ) µ Y X = The observations (X 1, Y 1 ),,(X n,y n ) are statistically independent. 立 7

28 Y ~ N( α + βx, σ ) N( α + β x, σ 1 ) N( α + β x, ) N( α + βx3, σ σ ) µ Y x = α + βx 8

29 Practically, only a sample data is collected and α, β, σ are unknown P(y) " µ Y x = α + βx"? : observations 9

30 How to estimate the regression equation using a sample data? Y " µ Y x = α + βx"? : observations 30

31 Estimation of the model Observations : (X 1, Y 1 ),,(X n,y n ) Regression model : Y X=x~ N( µ Y x = α + βx, σ ) Regression equation : µ Y x = α + βx Estimation : α =? β =? σ =? 31

32 Let a, b be estimates of α,β Predicted equation : Y = a + b x, it could be a 1. predicted value of Y : Y Ex. 若 X=60 數 Y X=60? µ Y x. estimated value of Ex. X=60 數 µ = =? Y X 60? µ Y x 3

33 Estimation of α,β A possible equation! Many possible equations! 33

34 34 observed Y; predicted Y Prediction error = Y-Y min imum 4 4 Y ') (Y 3 1 i i i = + + = = 44 6 Y ') (Y 3 1 i i i = + + = = Y ') (Y 3 1 i i i = + + = =

35 In the predicted equation, the intercept a =? The slope 率 b=? Least Squares estimates (LSE, ) a, b : Principle : find a regression equation which minimizes the sum of squared differences between the actual Y and the predicted Y min imize n i= 1 (Yi Y i ') 35

36 Formulae : Linear regression equation : Y = a + bx Estimated regression coefficients : b = S /S,a = Y bx xy x where S = (x x)(y y)/(n 1) = { xy nxy}/(n 1), xy S = (x x) /(n 1) = { x nx }/(n 1) x 36

37 Meaning of the estimated intercept, a a = Y at X=0. µ Y = The estimated value of X 0 when X=0. Example. X Y 零 金 0 零 金 = a The predicted value of Y when X=0. Example. X Y 零 金 若 0 零 金 a 0 不 數 X a Example. X= Y= X 不 0 a 37

38 a is an estimate of the true intercept. One may interest in testing H0 : =0. When =0, the equation passes through the origin( ), µ µ Y x Y x= 0 = βx = 0 38

39 Meaning of the estimated slope, b b = increment with unit change of x When there is one unit change in x, the increment/decrement in µ Y x Example. In previous case, if b=0., X 1 零 金 0. µ Y x 39

40 b is an estimate of the true slope. One is more interested in testing H0 : =0. When =0, the equation is a constant 數 and independent of X values, µ Y x = α,y ~ N( α, σ The distribution of Y is uncorrelated with X. X and Y are independent! ) P(y) Y µ Y x = α x1 x x3 X 40

41 Example. X= Y X Y mean s.d. variance S = ( xy nxy)/(n 1) xy = { (73.5)(70.)}/(10 1) = b= S /S xy x = / = a = y bx = = Ans. Y = X 41

42 EXCEL : 數 數 度 Model fitting 數 Model estimating 4

43 EXCEL : output 數 Note : The difference to previous calculation is due to rounding error. 數 a, b estimates of, SE(a), SE(b) t- testing the null hypothesis of =0, =0 t-value(a)=a/se(a), t-value(b)=b/se(b) p- corresponding p-values to t-value(a), t-value(b) p-value (a) =0.9511>0.05, not reject that =0 p-value (b) = 0.0<0.05, reject! 0 95% 95% confidence interval for, 43

44 After a, b are obtained, σ =? µ Y x = α + βx a + bx = Y' 44

45 The standard error of estimate : Variance σ : Dispersion of Y around the regression line The variation of the random error, Error = = : unobtainable Standard error of estimate : Use residuals to estimate error, Residual = = Y-Y : observable Standard error of estimate is defined by s y x (Y Y') = n n 1 (S n S where Sy : sample s.d. of Y, Sx : sample s.d. of X = Y µ Y x Y X b ) 45

46 The standard error of estimate : σ or S y x The random variation is unexplained by the regression line. Great random variation Small random variation 46

47 Example. X= Y : Y = X X Y Y-Y' Note : Residual = = Y-Y : observable Error = = Y µ Y x : unobtainable 47

48 Example. X= Y : Y = X s y x (Y Y') = n = = 16.9 = (n 1) (n ) (S y S x b ) = 9 8 ( (0.93) ) 16.3 The standard error of estimate is

49 EXCEL : 數 數 sy x 49

50 ESTIMATION & PREDICTION Confidence intervals and prediction intervals ESTIMATION: Q: At X=x, the mean value of Y, Point estimation, confidence interval µ Y = x? PREDICTION: Q:If an individual is drawn from the population of X=x, Y=? Point prediction, prediction interval µ Y x3 =? Y x=? 50

51 Confidence interval of µ Y x at X=x Confidence interval : At X=x, the mean value of Y, Point estimation : Y = a+bx µ Y x 100(1- )% confidence interval : Y' ± t(n, α / ) sy x 1 n + (x x) (n 1)s x 51

52 Example. X= Y : Y = X Q : 60 95% Ans. 1. Point estimation : Y = (60)= % confidence interval : Y' = n = 10,(x x) Y' ± t 57.33, t (n, α / ) (8,0.05) s =.306,s = ( ) y x 1 n 57.33± (x x) + (n 1)s 1 10 y x + = 16.9, = 18.5,s x (8.94) x = 8.94 = 57.33±

53 Prediction interval of Y at X=x Prediction interval : If draw an individual from the population of X=x, Y=? Prediction : Y = a + bx 100(1- )% prediction interval : Y' ± t(n, α / ) sy x 1+ 1 n + (x x) (n 1)s x 異數 53

54 Example. X= Y : Y = X Q : 若 X=60 95% Ans. 1. Point estimation : Y = (60)= % confidence interval : Y' = 57.33, t =.306,s = 16.9, (8,0.05) y x n = 10,(x x) = ( ) = 18.5,s = 8.94 x 1 (x x) Y' ± t s 1+ + n (n 1)s (n, α /) y x x ± = 57.33± (8.94) 54

55 RECALL : ANOVA-table Recall an ANOVA-table in Chapter 1. SStotal = (Y Y) D.f. = n-1 for n observations. MStotal = SS total/(n-1) SST = due to treatment = (Yj Y) (Between-group variation) j:treatment Yj = estimated mean of Y of treatment-j group D.f. = k-1 for k treatments MST=SST/(k-1) Does treatment effect exist? SSE = due to random error = D.f. = n k MSE = SSE/(n-k) SS total = SST + SSE (Y Y j ) (Within-group variation) 55

56 ANOVA Table Source of Variation Sum of Squares Degrees of Freedom Mean Square Treatment SST k-1 SST/(k-1)=MST Error SSE n-k SSE/(n-k)=MSE Total SS total n-1 F MST/MSE F-test : H0 : No treatment effect exists Vs H1 : There are treatment effects. 56

57 In a regression model, SStotal = Total variation of Y : SStotal = Σ(Y Y) = (Y' Y) + (Y Y') = SSR + SSE SSR = The variation explained by the regression model SSE=The unexplained variation 57

58 SStotal= D.f. = n-1 for n observations. Mstotal = SS total/(n-1) In a regression model, ( Y Y) = (n 1)S SSR = due to regression model Y = estimated mean of Y at some X-level D.f. = -1=1 for regression coefficients MSR=SSR/1 = SSR Does the X-effect exist? Y = ( Y' Y) = (n Does the linearity exist? 1)b S X SSE = due to random error D.f. = n MSE = SSE/(n-) = S y x = ( Y Y') = (n )Sy x 58

59 ANOVA Table Source of Variation Sum of Squares Degrees of Freedom Mean Square Regression SSR -1 SSR/1=MSR Error SSE n- SSE/(n-)=MSE Total SS total n-1 F MSR/MSE F-test : The regression line is horizontal. vs. H0 : No global linear effect exists =0 H1 : There is a linear association. 59

60 (X) (Y) sum mean sd variance 60

61 度 SS total = SS reg = (Y Y) (Y' Y) = (n 1)S = (n 1)b SS error = SS total SS reg = Further, since for F-test, p-value = 0.0< 0.05, the linearity exists. Y S = X =

62 The Coefficient of Determination Coefficient of Determination : 數 the proportion of the total variation of Y that is explained by the variation of X. 數 Y 度 來 數 X 例 數 Y 度 數 X 例 6

63 The Coefficient of Determination Coefficient of determination : SSR r = SStotal exp lained by mod el = total total un exp lained = total (Y' Y) = (Y Y) SSE = 1 SStotal Coefficient of determination = (correlation coefficient) 63

64 數 數 度 r = = = 0.51 = S S y x y x SSR.5 (0.7) SStotal = MSE = SSE /(n ) = , = = 16.9 Note:adj(r ) = 1 MSE MStotal = / 9 =

65 EXCEL I. Correlation analysis Scatter plot XY Correlation : 數 若 兩 數 CORREL 數 若 兩 數 料 數 II. Linear regression : 料 65

66 Exercise. Correlation analysis : 37, 39, 43 Linear regression analysis : 45, 46, 53, 57 EXCEL: 47, 49 66

67 Bonus(%) : by EXCEL Exercise X= 數 ; Y= 金 數 (X) 金 (Y) 1. (correlation analysis) Scatter plot, correlation matrix X Y ( =0.05). 立 (regression analysis) 1.. (predicted equation) 說 數 數 ( =0.05) 3. ANOVA 論 67