Jan/100
Can I Get Rid of This Parameter? (Statistical tests)
I'm working on a simulation that has a bunch of parameters. It's getting a bit complicated, so I'm on a witch-hunt for unimportant parameters. The suspect in question is 'channel length.' The channel length can take any non-negative integer value. The ultimate output of the simulation is a random variable, called the 'turn taking.' (More on the details of turn taking coming soon to a journal near you.)
So the question is: does the channel length affect the turn taking?
If you're totally up to scratch on statistical tests, then you probably won't learn much from this post. It's actually mostly a reminder to my future self for when I need to remember how to do this same thing agin.
The first statistical test that I employed was drawing a graph:
For that it looks kinda like maybe turn taking is not related to the channel length. But even averaging 30 independent runs for each channel length has left me with means that are still slightly different. How worried about those small differences should I be?
There are two different analytical statistical tests that I could use: the Mann-Whitney U test, or Student's t-test. Student's t-test assumes the data are normally distributed, which is usually not exactly the case. But for large sample sizes, a t-test is robust to small-ish deviations from a normal distribution.
The Mann-Whitney U test has the advantage of not assuming a normal distribution, but it the downside that it assumes the variances in the two groups are approximately equal. For some varieties of the t-test, the same assumption is made. The Mann-Whitney U test also has the significant pragmatic downside in that it's less common, and so it's less likely that people will be able to follow your paper if you use it.
With a t-test, you start with the null hypothesis that two distributions are the same. Then you calculate a t statistic value and a number of degrees of freedom using the equations from Wikipedia. I fooled around with the different possible assumption sets, but I settled on assuming that my equally sized samples had equal variance, mostly because Python's scipy.stats.ttest_ind() function made those assumptions.
Once you have a t value and n degrees of freedom, you calculate a one-sided p-value. A one-sided p-value is taken by integrating the t distribution probability density (with n degrees of freedom) from the t value you calculated until infinity. You can do this in Python with scipy.stats.t.sf(t, degrees_of_freedom). If you want a two-sided p-value, multiply the one-sided p-value by two. You'd use a one-sided p-value if you wanted to know if one of the distributions had a greater mean than the other, but use a two-sided p-value if you just want to know if they're different. scipy.stats.ttest_ind() will actually calculate a two-sided p-value for your along with a t-value.
I computed the t-test two-sided p-values comparing the sample with a channel length of 1 to the other 7 channel lengths in the graph:
1 vs 2: 0.07
1 vs 3: 0.11
1 vs 4: 0.15
1 vs 5: 0.13
1 vs 6: 0.17
1 vs 7: 0.18
1 vs 8: 0.01
The p-values are the chance that you would get values at least as different as you did if you assume the null hypothesis is true. So it's reasonably likely that the data for a channel length of 1 come from a different distribution than the data for a channel length of 8. But there is a 1% chance that they do both have the same mean.
In this case, even if the channel length makes a little difference, I'm going to get rid of it. But maybe not in another case.
Peter