Hw5solutions - Helps in Practicals PDF

Preview

Summary

Helps in Practicals...

Description

6.434J/16.391J Statistics for Engineers and Scientists

Apr 11

MIT, Spring 2006

Handout #13 Solution 5

Problem 1: Let a > 0 be a known constant, and let θ > 0 be a parameter. Suppose X1 , X2 , . . . , Xn is a sample from a population with one of the following densities. (a) The beta, β(θ, 1), density: fX (x | θ) = θxθ−1 ,

for 0 < x < 1. a

(b) The Weilbull density: fX (x | θ) = θax a−1 e−θx , (c) The Pareto density: fX (x | θ) =

θ

θa , x(θ+1)

for x > 0.

for x > a.

In each case, find a real-valued sufficient statistic for θ. Solution Let X  (X1 , X 2 , . . . , Xn ) be a collection of i.i.d. random variables Xi ’s, and let x  (x1 , x2 , . . . , x n ) be a collection of observed data. (a) For any x, the joint pdf is  θ n (x1 x2 · · · xn )θ−1 , if ∀i, 0 < xi < 1 fX (x | θ) = 0, otherwise;

= θ n (x1 x2 · · · xn )θ−1 × I(0,1) (x1 )I(0,1) (x2 ) · · · I(0,1) (xn ) .       g(T (x) | θ)

h(x)

Factorization theorem implies that

T (x)  x1 x2 · · · xn is a sufficient statistic for θ. (b) For any x, the joint pdf is  θ n an (x1 x2 · · · xn )a−1 e−θ fX (x | θ) = 0, = θ n e−θ

n i=1

n i=1

xai

, if ∀i, xi > 0; otherwise;

xai

g(T (x) | θ)



× an (x1 x2 · · · xn )a−1 

×I(0,∞) (x1 )I(0,∞) (x2 ) · · · I(0,∞) (xn ) .   h(x)

1

Factorization theorem implies that T (x) 

n 

xai

i=1

is a sufficient statistic for θ. (c) For any x, the joint pdf is  θn anθ  , if ∀i, xi > a; (x1 x2 ···xn )θ+1 fX (x | θ) = 0, otherwise; =

θ n anθ ×I (x1 )I(a,∞) (x2 ) · · · I(a,∞) (xn ) . (x1 x2 · · · xn )θ+1 (a,∞)      h(x)

g(T (x) | θ)

Factorization theorem implies that

T (x)  x1 x2 · · · xn is a sufficient statistic for θ. Problem 2: a) Let X1 , X2 , . . . , Xn be independent random variables, each uniformly distributed on the interval [−θ, θ], for some θ > 0. Find a sufficient statistic for θ. b) Let X1 , X2 , . . . , Xn be a random sample of size n from a normal N (θ, θ ) distribution, for some θ > 0. Find a sufficient statistic for θ. Solution a) For any x  (x1 , x 2 , . . . , xn ), the joint pdf is given by   n  1 , if ∀i, −θ ≤ xi ≤ θ; 2θ fX (x | θ) = 0, otherwise;   n  1 , if −θ ≤ min(x1 , . . . , x ) and max(x , . . . , x ) ≤ θ; n 1 n 2θ = 0, otherwise;  1 n = 1 . I[−θ,∞) (min(x1 , . . . , x n ))I(−∞,θ] (max(x1 , . . . , xn )) ×   2θ   h(x) g(T(x) | θ)

2

Factorization theorem implies that   T(x)  min(x1 , . . . , xn ), max(x1 , . . . , xn )

is jointly sufficient for θ.

b) For any x  (x1 , x2 , . . . , xn ), the joint pdf is given by  1 n n 2 1 fX (x) = √ e− 2θ i=1 (xi −θ) 2πθ  1 n 2 2 1 n n = √ e− 2θ ( i=1 xi −2θ i=1 xi +nθ ) 2πθ  1 n n n 1 nθ x2 = √ e− 2θ i=1 i + i=1 xi − 2 2πθ  1 n  1 n n n 2 1 nθ = √ e i=1 xi × √ e− 2θ i=1 x i − 2 . 2π θ       h(x)

g(T (x) | θ)

Factorization theorem implies that

T (x) 

n 

x2i

i=1

is a sufficient statistic for θ. Problem 3: Let X be the number of trials up to (and including) the first success in a sequence of Bernoulli trials with probability of success θ, for 0 < θ < 1. Then, X has a geometric distribution with the parameter θ: Pθ {X = k} = (1 − θ)k−1 θ,

k = 1, 2, 3, . . . .

Show that the family of geometric distributions is a one-parameter exponential family with T (x) = x. [Hint : xα = eα ln x , for x > 0.] Solution Recall that the pmf of a one-parameter (θ) exponential family is of the form p(x | θ) = h(x) e

η(θ)T (x)−B(θ)

,

where x ∈ X . Rewriting the pmf of a Geometric random variable yields Pθ {X = x} = e (x−1) ln(1−θ)+ln θ

= ex ln(1−θ)−(ln(1−θ)−ln θ) ,

3

where x ∈ {1, 2, 3, . . . }. Thus, the geometric distribution is a one-parameter exponential family with h(x)  1

η(θ )  ln(1 − θ )

T (x)  x

B(θ)  ln(1 − θ) − ln θ

X  {1, 2, 3, . . . }.

Problem 4: Let X1 , X 2 , . . . , Xn be a random sample of size n from the truncated Bernoulli probability mass function (pmf),   p, if x = 1; P {X = x | p} =  (1 − p), if x = 0.

(a) Show that the joint pmf of X1 , X2 , . . . , X n is a member of the exponential family of distribution. (b) Find a minimal sufficient statistic for p.

Solution (a) Let x  (X1 , X2 , . . . X n ) denote the collection of i.i.d. Bernoulli random variables. The joint pmf is given by      P {X = x | p} = px1 (1 − p)1−x1 px2 (1 − p)1−x2 · · · pxn (1 − p)1−xn =p

n i=1

= e(ln p)

xi

(1 − p)n−

n i=1

xi

n i=1

xi

e[ln(1−p)][n−

= e[ln p−ln(1−p)]

n i=1

n i=1

xi +n ln(1−p)

xi ]

,

for x ∈ {0, 1}n . Therefore, the joint pmf is a member of the exponential family, with the mappings: θ=p

h(x) = 1

η(p) = ln p − ln(1 − p)

T (x) =

n 

xi

i=1

X = {0, 1}n .

B(p) = −n ln(1 − p)

(b) Let x, y ∈ {0, 1}n be given. Consider the likelihood ratio, P {X = x | p} = e [ln p−ln(1−p)][ P {X = y | p} 4

n i=1

xi −

n i=1

yi ]

.

Define a function k(x, y)  h(x)/h(y) = 1, which is bounded and non-zero for any x ∈ X and y ∈ X . n n Note that x and y such that i=1 xi = i=1 yi are equivalent because function k(x, y) satisfies the requirement of likelihood ratio partition. n Therefore, T (x)  i=1 xi is a sufficient statistic.

Problem 5: Let X1 , X 2 , . . . , Xm and Y1 , Y2 , . . . , Yn be two independent samples from N (µ, σ 2 ) and N (µ, τ 2 ) populations, respectively. Here, −∞ < µ < ∞, σ 2 > 0, and τ 2 > 0. Find a minimal sufficient statistic for θ  (µ, σ 2 , τ 2 ).

Solution Let X  (X1 , X 2 , . . . , Xm ) and Y  (Y1 , Y 2 , . . . , Yn ) denote the collections of random samples. The joint pdf (of Xj ’s and Yi ’s), evaluated at x  (x1 , x 2 , . . . , xm ) and y  (y1 , y 2 , . . . , yn ), is given by m (x −µ)2   1 m (yi −µ)2 j=1 j 1 n − ni=12τ 2 2σ2 · √ fX,Y (x, y | θ) = √ ·e · e− 2πτ 2 2πσ 2 µ 2 2 2 2 µ 1 1 m n m n = e− 2σ2 j=1 xj − 2τ 2 i=1 y i + σ2 j=1 xj + τ 2 i=1 yi −B(µ,σ ,τ ) , 2

2

ln 2πσ 2 + n2 ln 2πτ 2 + mµ + nµ . where B(µ, σ 2 , τ 2 )  m 2σ 2 2 2τ 2 Notice that the joint pdf belongs to the exponential family, so that the minimal statistic for θ is given by T(X, Y) 

m 

Xj2,

j=1

n  i=1

Yi2 ,

m  j=1

Xj ,

n  i=1

 Yi .

Note: One should not be surprised that the joint pdf belongs to the exponential family of distribution. Recall that Gaussian distribution is a member of the exponential family of distribution and that random variables, Xi ’s and Yj ’s, are mutually independent. Thus, their joint pdf belongs to the exponential family as well. Note: To derive the minimal sufficient statistic, one may alternatively consider likelihood ratio partition. The set D0 is defined to be    D0  (x, y) ∈ Rm+n  for all µ, for all σ 2 > 0, for all τ 2 > 0    fX,Y x, y | µ, σ 2 , τ 2 = 0 =∅

(empty set).

5

Let (x, y) ∈ / D0 and (v, w) ∈ / D0 be given. Their likelihood ratio is given by  n m n m 1   2  2 1  2  2  fX,Y (x, y | θ ) wi yi − vj − 2 xj − = exp − 2 fX,Y (v, w | θ) 2τ i=1 2σ j=1 i=1 j=1 +

n m n m     µ  µ  w . + v y − x − i j i j τ 2 i=1 σ 2 j=1 i=1 j=1

By definition, (x, y) ∈ / D 0 and (v, w) ∈ / D0 are equivalent iff there exists a function, 0 < k(·, ·, ·, ·) < ∞, which is independent of θ, such that fX,Y (x, y | θ) = k(x, y, v, w). fX,Y (v, w | θ) / D0 are equivalent if The likelihood ratio implies that (x, y) ∈ / D 0 and (v, w) ∈ and only if m 

x2j = yi2 = xj =

j=1

n 

(1)

n 

w2i ,

(2)

vj , and

(3)

wi ,

(4)

i=1 m

i=1 m



vj2 ,

j=1

j=1

n 

m 



j=1

yi =

i=1

n  i=1

where function k(x, y, v, w)  1. That is, (x, y) and (v, w) are in the same equivalent class iff conditions (1)-(4) are satisfied. Then a representation of the equivalent class is given by T(X, Y) 

m  j=1

Xj2,

n 

Yi2 ,

i=1

m  j=1

Xj ,

n  i=1

 Yi .

Thus, we have a minimal sufficient statistic, T(X, Y). Problem 6:

The two hypotheses about the probability density fX (x) of an

observed random variable X are H1 : H0 :

1 −|x| e , 2 1 2 1 fX (x) = √ e−2 x , 2π fX (x) =

6

for any x for any x.

(a) Find the likelihood ratio Λ(x). (b) The test is of the form H1

Λ(x) ≷ η . H0

Compute the decision regions for various values of the threshold η. Solution (a) Let x ∈ R denote an observation. The likelihood ratio is given by Λ (x) 

fX | H (x | H1 ) . fX | H (x | H0 )

Substituting the densities of random variable X (under hypothesis H1 and under hypothesis H0 ) yields the likelihood ratio   1 −|x| e 2  Λ (x) =  1 − 21 x2 √ e 2π  π 21 x2 −|x| . e = 2 (b) The decision region for hypothesis H1 , R1 , is the set of points x’s that give rise to the output decision H1 : R1  {x | the test decides H1 on input x} = {x | Λ (x) > η}. Similarly, the decision region for hypothesis H0 , R0 , is given by R0  {x | the test decides H0 on input x} = {x | Λ (x) ≤ η} = R\R1 ,

where the symbol “\” denotes the set difference. Substituting the expression of the likelihood ratio from part (a) yields the following definition of decision region R1 :     π 1x2 −|x| >η . e2 R1 = x  2

When 0 ≥ η, we will have R1 = R since ey > 0 ≥ η for any y. Thus, we will consider the case when η > 0. 7

Taking natural log both sides of the inequality and writing x2 as |x|2 yields      η √2 1 2 R1 = x  |x| − |x| − ln √ >0 . 2 π  √  When 1 + 2 ln η√π2 < 0, the decision region is empty (since the term,

b2 − 4ac, in the square root of the quadratic formula is negative).  √  π −1 e 2 , we will have a When 1 + 2 ln η√π2 ≥ 0, or equivalently, η ≥ 2 non-empty decision region,     √2   or R1 = x  |x| > 1 + 1 + 2 ln η π   √2 |x| < 1 − 1 + 2 ln η π  √     2  = x  |x| > 1 + 1 + 2 ln η π    = x  x > 1 +



(absolute value cannot be negative)  √2  1 + 2 ln η or π   √2 x < −1 − 1 + 2 ln η . π

Therefore, the decision region R1 is given by    R, for η ≤ 0;      ∅, for 0 < η < π2 e−1/2 ;  R1 = √ √  (1 + 1 + 2 ln(η 2/ π), ∞)      √   (−∞, 1 − 1 + 2 ln(η 2/√π) ), for  π e−1/2 ≤ η, 2

while the decision region R0 is given by

R0 = R\R1   ∅, for η ≤ 0;       R, for 0 < η < π2 e−1/2 ;   = √ √  1 + 2 ln(η 2/ π),   1−       1 + 1 + 2 ln(η √2/√π) , for  π e−1/2 ≤ η. 2 8...