Chi squared test

Question

I have written code in MATLAB for a Chi-Square test. I wish to obtain P-values as 0.897 or 0.287 and so on, but my results are too small. Below is my code:

pd = fitdist(sample, 'weibull');
[h,p,st] = chi2gof(sample,'CDF',pd)

I've also tried using the AD test with similar result:

dist = makedist('Weibull', 'a',A, 'b',B);
[h,p,ad,cv] = adtest(sample, 'Distribution',dist)

Below is a histogram of the data with a fitted Weibull density function (Weibull parameters are A=4.0420 and B=2.0853)

Answer 1

When the p-value is less than a predetermined significance level (default is 5% or 0.05), it means that the null hypotheses is rejected (which in your case means that the sample did not come from a Weibull distribution).

The chi2gof function first output variable h denotes the test result, where h=1 means that the test rejects the null hypothesis at the specified significance level.

Example:

sample = rand(1000,1);           % sample from Uniform distribution
pd = fitdist(sample, 'weibull');
[h,p,st] = chi2gof(sample, 'CDF',pd, 'Alpha',0.05)

The test clearly rejects H0, and concludes that the data did not came from a Weibull distribution:

h =
     1             % 1: H1 (alternate hypo), 0: H0 (null hypo)

p =
   2.8597e-27      % note that p << 0.05

st = 
    chi2stat: 141.1922
          df: 7
       edges: [0.0041 0.1035 0.2029 0.3023 0.4017 0.5011 0.6005 0.6999 0.7993 0.8987 0.9981]
           O: [95 92 92 97 107 110 102 95 116 94]
           E: [53.4103 105.6778 130.7911 136.7777 129.1428 113.1017 93.1844 72.8444 54.3360 110.7338]

---

Next let's try that again with a conforming sample:

>> sample = wblrnd(0.5, 2, [1000,1]);   % sample from a Weibull distribution

>> pd = fitdist(sample, 'weibull')
pd = 
  WeibullDistribution

  Weibull distribution
    A = 0.496413   [0.481027, 0.512292]
    B =  2.07314   [1.97524, 2.17589]

>> [h,p] = chi2gof(sample, 'CDF',pd, 'Alpha',0.05)
h =
     0
p =
    0.7340

the test now clearly passes with a high p-value.

---

EDIT:

Looking at the histogram you've shown, it does look like the data follows a Weibull distribution, although there might be cases of outliers (look at the right side of the histogram), which might explain why you are getting bad p-values. Consider preprocessing your data to handle extreme outliers..

Here is an example where I simulate outlier values:

% 5000 samples from a Weibull distribution
pd = makedist('Weibull', 'a',4.0420, 'b',2.0853);
sample = random(pd, [5000 1]);
%sample = wblrnd(4.0420, 2.0853, [5000 1]);

% add 20 outlier instances
sample(1:20) = [rand(10,1)+15; rand(10,1)+25];

% hypothesis tests using original distribution
[h,p,st] = chi2gof(sample, 'CDF',pd, 'Alpha',0.05)
[h,p,ad,cv] = adtest(sample, 'Distribution',pd)

% hypothesis tests using empirical distribution
[h,p,st] = chi2gof(sample, 'CDF',fitdist(sample,'Weibull'))
[h,p,ad,cv] = adtest(sample, 'Distribution', 'Weibull')

% show histogram
histfit(sample, 20, 'Weibull')

% chi-squared test
h =
     1
p =
    0.0382
st = 
    chi2stat: 8.4162
          df: 3
       edges: [0.1010 2.6835 5.2659 7.8483 25.9252]
           O: [1741 2376 764 119]
           E: [1.7332e+03 2.3857e+03 788.6020 92.5274]


% AD test
h =
     1
p =
   1.2000e-07
ad =
   Inf
cv =
    2.4924

The outliers are causing the distribution tests to fail (null hypothesis rejected). Still I couldn't reproduce getting a NaN p-value (you might wanna check this related question on Stats.SE about getting NaN p-values)..