A Biased Maximum Likelihood Estimator

Suppose $X_1, \ldots , X_n$ are uniformly distributed on $[0, \theta ]$, where $\theta$ is unknown. The maximum likelihood estimator for $\theta$ is $T = \max(X_1, ..., X_n)$. But the expected value of $T$ is $n\theta/(n+1)$. Let us see what this means. We will take $n=3$, to get an idea. Let us first select a $\theta$.

theta = 1 / rand(1,1);

Let us pretend we do not know the value of $\theta$. Using this 'unknown' $\theta$, we produce $n$ observations from a uniform distribution over $[0, \theta ]$.

n = 3;
x = theta*rand(1,n)

x =

    1.1118    0.1559    1.1211

For this realization, the estimate of $\theta$ is given as.

T = max(x)

T =

    1.1211

Let us check if T is close to $n\theta/(n+1)$.

T - n*theta/(n+1)

ans =

    0.2005

This is far from zero. But we said that the expected value of $T$ is $n\theta/(n+1)$. What's going on?

Here's an explanation : We said the expected value of $T$ is $n\theta/(n+1)$ but we just compared a single realization of $T$ to its expected value. To better estimate the expected value of $T$, we need to average over many realizations. The code below makes it more concrete.

K = 100; % number of realizations
AvgT = 0;
for i = 1:K,
    x = theta*rand(1,n);
    T = max(x);
    AvgT = AvgT + T/K;
end
AvgT - n*theta/(n+1)

ans =

   -0.0154

This is much closer to zero. If we increase the number of realizations, the average of $T$ further approaches its expected value.

K = 10^5; % number of realizations
AvgT = 0;
for i = 1:K,
    x = theta*rand(1,n);
    T = max(x);
    AvgT = AvgT + T/K;
end
AvgT - n*theta/(n+1)

ans =

   3.2041e-04

Note that this discussion is justified by the Weak Law of Large Numbers.

Ilker Bayram, Istanbul Teknik Universitesi, 2015