What Would It Take???

[New2vp]

A search for something better
Frank, I don't have the time to post that I used to have, so I don't know if I can make further timely responses to questions.  In one of the cases below, I can agree with you but in most of these cases, I believe you are being misled by your intuition.  The Pearson chi-squared test is suggested, chosen, and studied, because of its optimality conditions as a hypothesis test.  Optimality conditions aren't taught in elementary stat courses and aren't often well understood in intermediate courses.  In layman's terms, a test is optimal if, for a given probability of a false positive like 5%, it has the best chance of detecting that the null hypothesis is false when it in fact is false.  OK, I admit that was not exactly layman's terms!  There is something called a Neyman-Pearson lemma if you would like to research that.  (The Pearson who came up with the chi-squared test is the father (Karl) of the Pearson who is named in the lemma (Egon).  No word on whether Egon was suffering from confirmation bias in trying to justify that his father had come up with a form of a "best" test. )There's several reasons why I've decided I don't like The Chi Squared Test for VP analysis.1. It treats things with extremely divergent frequencies more or less the same. Intuition tells me that flipping a coin twenty times and expecting it to come up heads 10 times is fundamentally different from playing 350,000 hands of video poker and expecting 10 Royals. The Chi test only looks at expected vs. actual and seems to be somewhat frequency irrespective. Expecting 10 heads on twenty flips of a 50/50 coin toss, simply isn't the same as expecting 10 Royals on a 1 in 35,000 probability. To Chi they are the same.Let me try to help your intuition.  When an intuition is informed better, different things become "intuitive."  What you're seeing is a bit of an illusion.  The components of the chi-squared test (the numbers that you add up to get the test statistic) are all fractions in which the numerator is a squared deviation and the denominator is an expected value for a particular cell in your table.  Thus the things you are adding up are in a sense all normalized as if they were percentages.  So checking to see if heads comes up 50% of the time is not all that different from checking to see if royals come up 0.0029% of the time.  We are still checking a percentage rather than the absolute number.Instead of squared deviations in the numerator, you could have unsquared deviations or the absolute value of deviations, but statisticians like to use squared deviations (variances instead of standard deviations) because they have nice mathematical properties (like differentiability), not to mention meeting optimality conditions of hypothesis testing.  Differentiability, the ability to take a derivative at all points is helpful in calculus for optimization, being able to find maxima and minima. What would be a problem is checking for the number of royals that come up in only 1000 hands.  First of all it would not be unusual to have 0 royals which is the lowest possible result, so the test would be unable to really prove anything.  The other problem (when the expected value of any cell is too low) is not the chi-squared test or test statistic itself, but the fact that the distribution of the test statistic (under the assumption that the null is true) will no longer close enough to the chi-square distribution to be able to use relatively simple formulas to calculate the critical value (the number which will allow you to reject a hypothesis of randomness).  But I don't think any test is going to be very good at rejecting a null hypothesis involving the number of royals unless you have more than a few hundred thousand hands.2. If casinos were going to go to the trouble of gaffing machines, we can say with almost absolute certainty that they would NOT gaff them in the players favor. The Chi test treats getting too many of something exactly the same as getting too few. This isn't realistic. Getting too many of something should reduce our skepticism and getting too few would increase it.I have less argument with this issue than the others.  Certainly I agree that intentional gaffing should be in the casino's favor.  Trying to find a test that is sensitive only to one-way deviations is not unheard of.  One thing that I could think of is if you want a 5% test, perform a 10% chi-square test and only reject randomness if the largest component of your test statistic arises from a deviation that would be against the player.  This would then be a 5% test consistent with your concern, but I could not vouch for the optimality conditions of this new made-up test.Another concern for those who use the test is that they might simply believe that the RNG isn't sufficiently random, without intentional gaffing.  Not all RNG's are "good."  See Diehard tests by George Marsaglia.  There are plenty of examples where random number generators in the past had cycles or problems in actually behaving sufficiently randomly.  I have no special knowledge whether or not vp-machine RNG's pass these tests, but these problems have been well-known by mathematicians for quite a few years now, so I would suspect that the people putting them in SHOULD know enough to use good, predictable RNGs.3. If you track enough things, like flushes, full houses, etc...You would expect to run good on some and bad on others. And indeed the more things you tracked the more askew you'd expect your distribution to look. The Chi Test is blind to this kind of normal variance. If one did run extremely good on flushes and extremely bad on Full Houses the Chi Test would render a larger chance of the machines being non-random when in fact I'd say that type of fluctuation is exactly what one should expect.Here you are neglecting the idea that if you are measuring more categories, the test statistic and the critical value will both be higher because you will be selecting a higher number of "degrees of freedom."  The Chi-squared test accounts for the number of things that you track; it is not blind to an increased number of categories.4. Getting exactly the number of things you'd expect to get is just as unlikely as getting too few or too many in a truly random distribution. The Chi test doesn't account for this at all. On class II machines that are clearly non-random and spit out hands based on a pre-determined pay rate, the chi test would find the abnormal consistancy to be proof of randomness when in fact it would be proof against.No, this is not true either, but I will say that it is lesser known how to deal with it.  However, it is relatively easy to understand.  The chi-squared test will suggest non-randomness if the results are too far away from expected.  It does this by producing too big of a test statistic.  However, when someone who is inexperienced "fudges" data that is supposed to be random, he often makes up data that is "too close" to expected.  In fact, that can and IS detected by chi-squared tests by the production of a test statistic that is too close to zero.  It may not be "well-known" but the famous genetic experiments of Gregor Mendel that posited dominant and recessive genes have been shown to be "too close" to what is expected.  Of course, the statistical tests to suggest that the monk's experiments were too good to be true didn't exist until after his death, so he probably thought he got away with it.  I've heard theories put forth that his numbers had to be too good to change opinions back in the day when scientific discovery was akin to blasphemy; or maybe he had some divine help.  Whatever, it is easy for chi-squared tests to be used to determine if the data is "too good to be true."  Simply use the other end of the distribution.  Instead of looking to see if the test statistic is bigger than the 99th percentile of the chi-squared distribution, check to see if it is smaller than the 1st percentile.I'm convinced that the Chi Test is far less than perfect for this task, what remains is for someone to recommend something better.The biggest part of the utility is the data tracking and entry. There's still plenty of time to add more and better tests. I hope we find something.~FK

[New2vp]

Frank, the upshot of the long post for those with short attention spans.  There may be something marginally better, but it will be quite difficult to find something that is better, easier for most to apply, and without other flaws.  Admittedly arguably, there may have been more flaws in your critique than you can find in the chi-squared test.

[Frank Kneeland]

Frank, the upshot of the long post for those with short attention spans.  There may be something marginally better, but it will be quite difficult to find something that is better, easier for most to apply, and without other flaws.  Admittedly arguably, there may have been more flaws in your critique than you can find in the chi-squared test.
I have little doubt that my critique was less than perfect. You also apparently didn't understand exactly what I was saying in some cases because your comments included things of which I was aware and considered, but didn't mention in the post because I was trying to keep it simple.The bottom line is that you feel Chi Squared is the best test we have and I'm taking your word on that without question.I should have it done by months end. After I post it you are welcome to clean it up and post your own version. With your math skills I'm sure "version New2" will become the definitive version everyone uses.~FK

[New2vp]

You also apparently didn't understand exactly what I was saying in some casesNo doubt, though I did try to understand.  A lot of my posts are long because I try both to give my opinion and then attempt to explain it or back it up.  And if you hadn't established a reputation of trying to get at that elusive thing that some call "truth" and exhibited the numerical knowledge that you have, I wouldn't have even attempted the explanation at trying to improve on what you were saying.As you explained in a different post, I predictably focused on the things that supported my argument and only focused on things that I thought you were wrong about, not the things that I thought you were correct in saying.  But then again, as nice as it feels when someone tells us we are correct, we sometimes learn more because of the things that we said or did that were wrong, at least in someone else's opinion.My goal however was not to prove that I was right and you were wrong.  I could have silently satisfied myself with that thought without posting and given myself an extra hour of sleep.  Instead, I hoped to help your understanding and give you a couple references to check rather than have to depend on some nameless, faceless blogger (me) and allow you to continue completing your utility with a clearer conscience about the properties of chi-squared tests.  I apologize if my normal inadequate ability at diplomacy overshadowed the intention to help.  (There are many "wronger" things posted here by others that I don't feel the need to spend the time to comment on; so I hope you in some way please see my attention here to your thread as a compliment.)Also, I think what you are doing is commendable and certainly more beneficial to this audience than the usual, "Is too! Is not!" debates that are increasingly tiresome to read.  Plus I think that will be the case whether or not anyone uses your test.  Most tools that are taught in quantitative classes for non-quantitative majors are not used by most students; but sometimes it is enough that people are at least faintly aware that the tools do in fact exist and that it is possible for someone to use them to perform such a test.  That alone will provide greater knowledge and understanding. 

[Frank Kneeland]

[QUOTE=Frank Kneeland]
You also apparently didn't understand exactly what I was saying in some casesNo doubt, though I did try to understand.  A lot of my posts are long because I try both to give my opinion and then attempt to explain it or back it up.  And if you hadn't established a reputation of trying to get at that elusive thing that some call "truth" and exhibited the numerical knowledge that you have, I wouldn't have even attempted the explanation at trying to improve on what you were saying.As you explained in a different post, I predictably focused on the things that supported my argument and only focused on things that I thought you were wrong about, not the things that I thought you were correct in saying.  But then again, as nice as it feels when someone tells us we are correct, we sometimes learn more because of the things that we said or did that were wrong, at least in someone else's opinion.My goal however was not to prove that I was right and you were wrong.  I could have silently satisfied myself with that thought without posting and given myself an extra hour of sleep.  Instead, I hoped to help your understanding and give you a couple references to check rather than have to depend on some nameless, faceless blogger (me) and allow you to continue completing your utility with a clearer conscience about the properties of chi-squared tests.  I apologize if my normal inadequate ability at diplomacy overshadowed the intention to help.  (There are many "wronger" things posted here by others that I don't feel the need to spend the time to comment on; so I hope you in some way please see my attention here to your thread as a compliment.)Also, I think what you are doing is commendable and certainly more beneficial to this audience than the usual, "Is too! Is not!" debates that are increasingly tiresome to read.  Plus I think that will be the case whether or not anyone uses your test.  Most tools that are taught in quantitative classes for non-quantitative majors are not used by most students; but sometimes it is enough that people are at least faintly aware that the tools do in fact exist and that it is possible for someone to use them to perform such a test.  That alone will provide greater knowledge and understanding. 
[/QUOTE]No I'm sorry I didn't elaborate on the exact issue I had that sparked the post. It's too long to get into. I appreciated all the advice and I agreed with what you said. And I'm more confident about using Chi now. You achieved all your goals.~FK

[Tedlark]

No I'm sorry I didn't elaborate on the exact issue I had that sparked the post. It's too long to get into. I appreciated all the advice and I agreed with what you said. And I'm more confident about using Chi now. You achieved all your goals.Actually Frank he hasn't run that sub 4 minute mile yet but there's still time.

[New2vp]

never got much under 6. these days I'm pretty sure I couldn't get under 12. you achieved a goal if that was to make me smile!

[Tedlark]

  That truely was my intention New, now I'm off for a run.

[moneyla]

Just checked in to see if it was soup yet? Nope. Is this a government contract? :-)

[Frank Kneeland]

Just checked in to see if it was soup yet? Nope. Is this a government contract? :-)You obviously didn't check anything but page 19, because on page 18, I stated exactly when it would be released.I'm coinciding it with my next month's BJI article.~FK