Unizor - Probability - Random Variables - Independence

There are a few different but equivalent approaches to define independent random variables. We will use a simple approach based on the fact that we are only dealing with random variables that take finite number of values.
So, assume a random variable ξ takes values
x1, x2,..., xM
with probabilities
p1, p2,..., pM.
Further, assume a random variable η takes values
y1, y2,..., yN
with probabilities
q1, q2,..., qN.

Let us remind that a random variable is a numeric function defined on each elementary event participating in the random experiment. The fact that a random variable ξ takes some value xi means that one of the elementary events, where the value of this numeric function equals to xi, indeed occurred. The combined probability of all the elementary events, where this numeric function equals to xi is the probability of a random variable ξ of having the value xi. The combination of all these elementary events make up an event characterized by a description "an event where random variable ξ takes the value xi".

For instance, when rolling two dice and summarizing the rolled numbers (this sum is our random variable), elementary events (1,5), (2,4), (3,3), (4,2) and (5,1) combined together form an event described as "our random variable took a value of 6".

Let's assign a symbol Ai to this combined event of a random variable ξ of taking the value xi. So, according to our notation,
P(ξ=xi) = P(Ai)

Analogously, let Bj be an event of a random variable η taking the value yj. So, according to our notation,
P(η=yj) = P(Bj)

The above considerations allow us to use the language and properties of events to describe the values of and relationships between random variables.
Thus, we can define a conditional probability of one random variable relative to another using already defined conditional probability between events:
P(ξ=xi | η=yj ) = P(Ai | Bj )

If the conditional probability of a random variable ξ taking any one of its values under the condition that a random variable η took any one of its values equals to an unconditional probability of ξ taking that value, then a random variable ξ is independent of a random variable η.
In other words, ξ is independent of η if
P(ξ=xi | η=yj ) = P(ξ=xi )
where index i can take any value from 1 to M and index j can take values from 1 to N.

Important Property of Independent Random Variables

From the definition and properties of conditional probability for events X and Y we know that
P(X | Y) = P(X∩Y)/P(Y)
Therefore,
P(ξ=xi | η=yj ) = P(ξ=xi ∩ η=yj ) / P(η=yj )
But, if random variable ξ and η are independent, the conditional probability of ξ taking some value under some condition imposed on η is the same as its unconditional probability. Therefore,
P(ξ=xi ) = P(ξ=xi ∩ η=yj ) / P(η=yj )
from which follows
P(ξ=xi ∩ η=yj ) = P(ξ=xi ) · P(η=yj )
Verbally, it can be expressed as the statement that the probability of two independent variable simultaneously taking some values equals to a product of probabilities of them separately taking their corresponding values.
This, for random variables, similarly to an analogous property of probability of independent events, is a characteristic property of their independence. It is completely equivalent to a definition of independent variables that uses conditional probability.

Unizor - Probability - Random Variables - Problems 2

Problem 2.1.
Calculate expected value E(ξ), variance Var(ξ) and standard deviation σ(ξ) of a random variable ξ equal to a winning in a simplified game of roulette played by the following rules:
(a) You bet $1 on a number 23 in a game of roulette with a spinning wheel that includes numbers from 1 to 36, 0 and 00.
(b) If the ball stops on a spinning wheel in a cell with a number 23, you win $36.
(c) If the ball stops in any other cell (including 0 and 00), you lose your $1 bet.

Answer:
E(ξ) = −1/38 ≅ −0.026316
Var(ξ) ≅ 35.078355
σ(ξ) ≅ 5.922690

Solution

Our first step is to specify the values
x1, x2,..., xN
that our random variable ξ takes, and the corresponding probabilities
p1, p2,..., pN
of each such value.

First of all, there are only two outcomes of this game, win or lose. Correspondingly, the number N of different values our random variable can take equals to 2. So, we are dealing with two random values, 36 (win) and −1 (lose).
Considering there are 38 different outcomes for a position of ball on the spinning wheel (numbers from 1 to 36, 0 and 00) with equal chances of each, the probability of winning is 1/38, while the probability of losing is 37/38.

Here is full specification of our random variable:
P(36) = 1/38
P(−1) = 37/38

The expected value of this random variable equals to a weighted average of its values with the corresponding probabilities as weights.
E(ξ) =
36·(1/38) + (−1)·(37/38) =
= (36−37)/38 = −1/38 ≅
≅ −0.026316

The variance is a weighted average of squares of differences between the values of our random variable and its expected value with corresponding probabilities as weights.
Var(ξ) =
[36−(−1/38)]^2·(1/38) +
+ [−1−(−1/38)]^2·(37/38) ≅
≅ 35.078355

Finally, the standard deviation is a square root of the variance and equals to
σ(ξ) ≅ 5.922690
The end.

Problem 2.2.
The problem above is a particular case of a more general problem. Assume the random variable that takes only two values, X and Y with probabilities P(X)=p and P(Y)=q, where both p and q are non-negative and p+q=1.
Determine its expectation, variance and standard deviation.

Answer:
E(ξ) = X·p+Y·q
Var(ξ) = (X−Y)^2·p·q
σ(ξ) = |X−Y|·√(p·q)

Solution

There are only two outcomes of this experiment with our random variable taking values X or Y with probabilities p and q correspondingly, where p+q=1 (we will use this equality to replace 1−q with p and 1−p with q).
So, the expectation equals to
E(ξ) = X·p+Y·q
The variance, as a weighted average of squares of differences between the values of our random variable and its expectation, where weights are probabilities, is equal to
Var(ξ) =
[X−(X·p+Y·q)]^2·p +
+ [Y−(X·p+Y·q)]^2·q =
= (X−Y)^2·q^2·p + (Y−X)^2·p^2·q =
= (X−Y)^2·p·q·(p+q) =
= (X−Y)^2·p·q
Finally, the standard deviation is a square root from the variance and equals to
σ(ξ) = |X−Y|·√(p·q)

Unizor - Probability - Random Variables - Problem 1

Calculate expected value E(ξ), variance Var(ξ) and standard deviation σ(ξ) of a random variable ξ formed by a sum of numbers on the top of two perfect dice rolled together.

Unizor - Probability - Random Variables - Variance

Random variables take many different values with different probabilities. Full description of these discreet random variables includes enumeration of all its values and corresponding probabilities. That is a full but lengthy description of a random variable. Though it is full, it does not provide an easy answer to such question as "What is my risk if I play this game for money?" or "What is an interval I should observe the values of this random variable most of the time?" There is, however, a desire to characterize the random variable with a small number of properties to give an idea of its behavior, evaluate risk (if risk is involved) and predict its values with certain level of precision, which, arguably, is one of the main purposes of theory of probabilities.

The first such property that we have introduced is the expected value or expectation of a random variable. Though it does provide some information about the random variable, it's not really sufficient to make good predictions about its values.

Assume that our random variable ξ takes values
x1, x2,..., xN
with probabilities
p1, p2,..., pN.
Then its expectation is
E(ξ) = x1·p1+x2·p2+...+xN·pN
This expectation can be viewed as a weighted average of the values of our random variables with probabilities of these values taken as weights .

To evaluate the deviation of the values of a random variable from its expectation, we are interested in weighted average of the squares of differences between values of this random variable and its expectation, that is in the expectation of a new random variable (ξ−a)^2, where a=E(ξ):
Var(ξ) = (x1−a)^2·p1+(x2−a)^2·p2+...+(xN−a)^2·pN

Unizor - Probability - Random Variables - Expectation Sum

Our goal in this lecture is to prove that expectation of a sum of two random variables equals to a sum their expectations.
Consider the following two random experiments (sample spaces) and random variables defined on their elementary events.

Ω1=(E1,E2,...,Em )
with corresponding measure of probabilities of these elementary events
P=(P1,P2,...,Pm )
(that is, P(Ei )=Pi - non-negative numbers with their sum equaled to 1)
and random variable ξ defined for each elementary event as
ξ(Ei) = Xi where i=1,2,...m

Ω2=(F1,F2,...,Fn )
with corresponding measure of probabilities of these elementary events
Q=(Q1,Q2,...,Qn )
(that is, Q(Fj )=Qj - non-negative numbers with their sum equaled to 1)
and random variable η defined for each elementary event as
η(Fj) = Yj where j=1,2,...,n

Separately, the expectations of these random variables are:
E(ξ) = X1·P1+X2·P2+...+Xm·Pm
E(η) = Y1·Q1+Y2·Q2+...+Ym·Qn

Let's examine the probabilistic meaning of a sum of two random variables defined on two different sample spaces.
Any particular value Xi+Yj is taken by a new random variable ζ=ξ+η defined on a new combined sample space Ω=Ω1×Ω2 that consists of all pairs of elementary events (Ei,Fj ) with the corresponding combined measure of probabilities of these pairs equal to
R(Ei,Fj ) = Rij
where index i runs from 1 to m and index j runs from 1 to n.

Thus, we have defined a new random variable ζ=ξ+η defined on a new sample space Ω of M·N pairs of elementary events from two old spaces Ω1 and Ω2 as follows
ζ(Ei,Fj ) = Xi+Yj
with probability Rij
Consider a sum Ri1+Ri2+...+Rin. It represents a probability of the first experiment resulting in a fixed elementary event ei while the second experiment resulting in either F1 or in F2, or in any other elementary event it may. That is, the result of the second experiment is irrelevant and this sum simply represents a probability of the first experiment resulting in Ei, that is it is equal to Pi:
Ri1+Ri2+...+Rin = Pi.
Similarly, fixing the result of the second experiment to Fj and letting the first experiment to end up in any way it may, we conclude
R1j+R2j+...+Rmj = Qj.
Keeping in mind the above properties of probabilities Rij, we can calculate the expectation of our new random variable ζ.
E(ζ) = E(ξ+η) =
= (X1+Y1)·R11+...+(X1+Yn)·R1n +
+ (X2+Y1)·R21+...+(X2+Yn)·r2n +
...
+ (Xm+Y1)·Rm1+...+(Xm+Yn)·Rmn =
(opening parenthesis, changing the order of summation and regrouping)
= X1·(R11+...+R1n ) +
+ X2·(R21+...+R2n ) +
...
+ Xm·(Rm1+...+Rmn ) +
+ Y1·(R11+...+Rm1 ) +
+ Y2·(R12+...+Rm2 ) +
...
+ Yn·(R1n+...+Rmn ) =
(using the properties of sums of combined probabilities Rij )
= X1·P1 +...+ Xm·Pm +
+ Y1·Q1 +...+ Yn·Qn =
= E(ξ) + E(η)
End of proof.

Unizor - Probability - Random Variables - Expectation Examples

Example 1
What is the expected value of a random variable ξ that equals to a sum of two numbers obtained by rolling two dice?
We have 36 combinations of numbers as a result of rolling two dice. Each pair has the same chances to occur as any other, therefore, the probability of each pair equals to 1/36.
The sum of two numbers takes values from 2 to 12 in the following manner:
2:1,1
3:1,2/2,1
4:1,3/2,2/3,1
5:1,4/2,3/3,2/4,1
6:1,5/2,4/3,3/4,2/5,1
7:1,6/2,5/3,4/4,3/5,2/6,1
8:2,6/3,5/4,4/5,3/6,2
9:3,6/4,5/5,4/6,3
10:4,6/5,5/6,4
11:5,6/6,5
12:6,6
Since each pair has a probability of 1/36 and, as we know, the probability is an additive measure, the probability of a combination of N pairs equals to N/36. Hence, the probabilities of different values for a sum of numbers on two dice equal to
P(ξ=2)=1/36
P(ξ=3)=2/36
P(ξ=4)=3/36
P(ξ=5)=4/36
P(ξ=6)=5/36
P(ξ=7)=6/36
P(ξ=8)=5/36
P(ξ=9)=4/36
P(ξ=10)=3/36
P(ξ=11)=2/36
P(ξ=12)=1/36
Using the formula for expected value
E(ξ) = p1·x1+p2·x2+...+pN·xN
We can calculate the expected value in this particular case as
E(ξ) = (1/36)·2 + (2/36)·3 +
+ (3/36)·4 + (4/36)·5 +
+ (5/36)·6 + (6/36)·7 +
+ (5/36)·8 + (4/36)·9 +
+ (3/36)·10 + (2/36)·11 +
+ (1/36)·12 =
= (2+6+12+20+30+42+
+40+36+30+22+12)/36 =
= 252/36 = 7
So, the expected value of a sum of two numbers obtained by rolling two dice equals to 7.
It means that, if we repeat our experiment with rolling of two dice a very large number of times, the average sum of these two numbers per experiment would be very close to 7.

Example 2
What is the average value of a card in the game of Blackjack (assume for simplicity that Aces are always valued at 11 points)?
The standard deck of cards contains 52 cards valued by numbers on the cards (from 2 to 10) or by 10 for pictures (Jack, Queen and King), or by 11 (Ace) with four suits per each. Therefore, we have
4 cards valued at 2 points
4 cards valued at 3 points
4 cards valued at 4 points
4 cards valued at 5 points
4 cards valued at 6 points
4 cards valued at 7 points
4 cards valued at 8 points
4 cards valued at 9 points
16 cards (10,J,Q,K) at 10 points
4 cards valued at 11 points
Since the probability to get each card is 1/52, the expected value of any card pulled randomly from a deck is
(4/52)·2+(4/52)·3+(4/52)·4+(4/52)·5+(4/52)·6+(4/52)·7+
+(4/52)·8+(4/52)·9+(16/52)·10+(4/52)·11 =
= (8+12+16+20+24+28+32+36+160+44)/52 = 380/52 ≅ 7.3
Of course, the situation above is a simplification of the real case. The rule that Ace can be counted as 1 or 11, as you wish, complicates the real picture.
Another complication is that usually a few cards have already been given to other players or yourself, so the probabilities must be based not on a full deck, but on whatever is left after the first few cards were distributed and depends on these cards.
However, Blackjack is usually played in a casino with more than one deck (like 5 or 6), which decreases the influence of previously dealt cards on the probabilities.

In conclusion, let's notice that the expected value of a random variable is not necessarily one of its real values. For instance, in Example 1 the expected value of 7 is the real value for two dice in positions 1+6, 2+5, 3+4, 4+3, 5+2 and 6+1. However, in Example 2 the expected value of 380/52≅7.3 cannot be a real value of the card.

Unizor - Probability - Random Variables - Expected Values

Assume our random experiment (like spinning the roulette wheel and a ball on it) results in the following K elementary events (like ball stops in a partition with some number):
Ω = { e1, e2,..., eK }
Assume further that we know the probabilities of each elementary event (maybe, they have equal chances, maybe not):
P(ei) = pi (i runs from 1 to K)
Finally, assume we have defined a random variable on this sample space that represents a numerical result of an experiment (like winning on a $1 bet):
ξ(ei) = xi (i runs from 1 to K)
If we conduct our experiment N times (consider this a very large number), we expect, approximately, that elementary event e1 will occur in N·p1 number of cases with our random variable, correspondingly, taking a value of x1.
Similarly, in, approximately, N·p2 number of cases our random experiment will end up at elementary event e2 and our random variable will take a value of x2.
And similar for all other indices.
Knowing this statistics, we can approximate the average value of our random variable ξ. If during N experiments in, approximately, N·p1 number of cases it took value x1, in, approximately, N·p2 number of cases it took value x2, etc. then the sum of all values in took in all experiments equals to
N·p1·x1 + N·p2·x2 + ... + N·pK·xK
and the average value of our random variable ξ per single experiment equals to
E(ξ) = p1·x1+p2·x2+...+pK·xK
This value depends only on probabilities of elementary events pi and corresponding values of our random variable xi and is called mathematical expectation of a random variable or just expectation or expected value.

Can we say that our random variable "takes an average value" calculated above as E(ξ)? No, this is not correct. It might never take this value. But, with the number of random experiments, where it is a numerical result, grows to infinity, its average value per experiment (that is, a sum of all values divided by the number of experiments) tends to E(ξ). The correct statement about this is that our random variable has an expected value (calculated above based on probabilities and individual values) E(ξ).