Definition
Intuitive definition
Cov(X,Y) measures the likelihood of X,Y “moving in the same direction”.
If I know X is less than expected, what does that tell me about Y?
- Nothing; Y is still expected to be E[Y]⇒Cov(X,Y)=0
- Y is also likely to be less than expected ⇒Cov(X,Y)>0
- Y is now likely to be more than expected ⇒Cov(X,Y)<0
- i.e., X,Y “tend to move” in different directions
Cov(X,Y)=E[(X−μ1)⋅(Y−μ2)]
Where μ1=E[X],μ2=E[Y]
Note that V(X)=Cov(X,X)
A simpler formula:
Cov(X,Y)=E[XY]−μ1μ2
Proof: Likely on the #ProbabilityExam
Cov(X,Y)Q.E.D=E[(X−μ1)⋅(Y−μ2)]=E[XY−μ1Y−μ2X+μ1μ2]=E[XY]−μ1μ2E[Y]−μ2μ1E[X]+μ1μ2=E[XY]−μ1μ2
Properties of Covariance
- Cov(aX,bY)=ab⋅Cov(X,Y)
- Cov(a+X,b+Y)=Cov(X,Y)
- For any X,Y: V(X+Y)=V(X)+V(Y)+2Cov(X,Y)
Proof of Property 3: Likely on the #ProbabilityExam
V(X+Y)=E[(X+Y)2]−((E[X](μ1+E[Y])+μ2)E[X+Y])2=E[X2+2XY+Y2]−(μ12+2μ1μ2+μ22)=E[X2]+2E[XY]+E[Y2]−(μ12+2μ1μ2+μ22)=(E[X2]−μ12)+(E[Y2]−μ22)+2(E[XY]−μ1μ2)=V(X)+V(Y)+2Cov(X,Y)Q.E.D.
Corollary of Property 3:
V(X−Y)=V(X+(−Y))=V(X)+V(Y)V(−Y)+2−Cov(X,Y)Cov(X,−Y)=V(X)+V(Y)−2Cov(X,Y)
Pearson Correlation Coefficient
Definition
ρ(X,Y) is a normalized version of Cov(X,Y).
−1≤ρ(X,Y)≤1 (Sign of ρ is the same sign of Cov)
- If ρ(X,Y) is close to 0, then the correlation between X,Y is weak
- If ρ(X,Y) is close to 1, then X,Y are strongly positively correlated
- X,Y have a strong tendency to move in the same direction
- If ρ(X,Y) is close to −1, then X,Y are strongly negatively correlated
- X,Y have a strong tendency to move in opposite directions
ρ(X,Y)=σ(X)⋅σ(Y)Cov(X,Y)