Probability· Note 09

Covariance

Year 2 · Semester ElulLast updated

Definition
Intuitive definition

Cov⁡(X,Y)\covariant(X,Y) measures the likelihood of X,YX,Y “moving in the same direction”.
If I know XX is less than expected, what does that tell me about YY?

  1. Nothing; YY is still expected to be E[Y]⇒Cov⁡(X,Y)=0E[Y] \Rightarrow \covariant(X,Y)=0
  2. YY is also likely to be less than expected ⇒Cov⁡(X,Y)>0\Rightarrow \covariant(X,Y)>0
  3. YY is now likely to be more than expected ⇒Cov⁡(X,Y)<0\Rightarrow \covariant(X,Y)<0
    • i.e., X,YX,Y “tend to move” in different directions
Formal definition
Cov⁡(X,Y)=E[(X−μ1)⋅(Y−μ2)]\covariant(X,Y)=E[(X-\mu_1)\cdot(Y-\mu_2)]

Where μ1=E[X],μ2=E[Y]\mu_1=E[X],\mu_2=E[Y]
Note that V(X)=Cov⁡(X,X)V(X)=\covariant(X,X)

A simpler formula:

Cov⁡(X,Y)=E[XY]−μ1μ2\covariant(X,Y)=E[XY]-\mu_1\mu_2

Proof: Likely on the #ProbabilityExam

Cov⁡(X,Y)=E[(X−μ1)⋅(Y−μ2)]=E[XY−μ1Y−μ2X+μ1μ2]=E[XY]−μ1E[Y]⏟μ2−μ2E[X]⏟μ1+μ1μ2=E[XY]−μ1μ2Q.E.D\begin{aligned} \covariant(X,Y)&=E[(X-\mu_1)\cdot(Y-\mu_2)] \\&=E[XY-\mu_1Y-\mu_2X+\mu_1\mu_2] \\&=E[XY]-\mu_1\underbrace{E[Y]}_{\mu_2}-\cancel{\mu_2\underbrace{E[X]}_{\mu_1}}+\cancel{\mu_1\mu_2} \\&=E[XY]-\mu_1\mu_2 \\Q.E.D \end{aligned}

Properties of Covariance

  1. Cov⁡(aX,bY)=ab⋅Cov⁡(X,Y)\covariant(aX,bY)=ab\cdot\covariant(X,Y)
  2. Cov⁡(a+X,b+Y)=Cov⁡(X,Y)\covariant(a+X,b+Y)=\covariant(X,Y)
  3. For any X,YX,Y: V(X+Y)=V(X)+V(Y)+2Cov⁡(X,Y)V(X+Y)=V(X)+V(Y)+2\covariant(X,Y)

Proof of Property 3: Likely on the #ProbabilityExam

V(X+Y)=E[(X+Y)2]−(E[X+Y]⏟(E[X]+E[Y])(μ1+μ2))2=E[X2+2XY+Y2]−(μ12+2μ1μ2+μ22)=E[X2]+2E[XY]+E[Y2]−(μ12+2μ1μ2+μ22)=(E[X2]−μ12)+(E[Y2]−μ22)+2(E[XY]−μ1μ2)=V(X)+V(Y)+2Cov⁡(X,Y)Q.E.D.\begin{aligned} V(X+Y)&=E[(X+Y)^2]-(\underbrace{E[X+Y]}_{\begin{aligned}(E[X]&+E[Y])\\(\mu_1&+\mu_2)\end{aligned}})^2 \\&=E[X^2+2XY+Y^2]-(\mu_1^2+2\mu_1\mu_2+\mu_2^2) \\&=E[X^2]+2E[XY]+E[Y^2]-(\mu_1^2+2\mu_1\mu_2+\mu_2^2) \\&=\textcolor{orange}{(E[X^2]-\mu_1^2)}+\textcolor{purple}{(E[Y^2]-\mu_2^2)}+\textcolor{blue}{2(E[XY]-\mu_1\mu_2)} \\&=\textcolor{orange}{V(X)}+\textcolor{purple}{V(Y)}+\textcolor{blue}{2\covariant(X,Y)} \\&Q.E.D. \end{aligned}

Corollary of Property 3:

V(X−Y)=V(X+(−Y))=V(X)+V(−Y)⏟V(Y)+2Cov⁡(X,−Y)⏟−Cov⁡(X,Y)=V(X)+V(Y)−2Cov⁡(X,Y)\begin{aligned} V(X-Y)&=V(X+(-Y))\\ &=V(X)+\underbrace{V(-Y)}_{V(Y)}+2\underbrace{\covariant(X,-Y)}_{-\covariant(X,Y)} \\&=V(X)+V(Y)-2\covariant(X,Y) \end{aligned}

Pearson Correlation Coefficient

Definition

ρ(X,Y)\rho(X,Y) is a normalized version of Cov⁡(X,Y)\covariant(X,Y).
−1≤ρ(X,Y)≤1-1\le\rho(X,Y)\le1 (Sign of ρ\rho is the same sign of Cov⁡\covariant)

  • If ρ(X,Y)\rho(X,Y) is close to 0, then the correlation between X,Y is weak
  • If ρ(X,Y)\rho(X,Y) is close to 11, then X,YX,Y are strongly positively correlated
    • X,YX,Y have a strong tendency to move in the same direction
  • If ρ(X,Y)\rho(X,Y) is close to −1-1, then X,YX,Y are strongly negatively correlated
    • X,YX,Y have a strong tendency to move in opposite directions
ρ(X,Y)=Cov⁡(X,Y)σ(X)⋅σ(Y)\rho(X,Y)=\frac{\covariant(X,Y)}{\sigma(X)\cdot\sigma(Y)}