What Would It Take???

[BillyJoe]

[QUOTE=money] I don't know if this is the best scientific method but I think the easiest test is the best test.

If the video poker game is fair, shouldn't all cards have an equal chance of being dealt?  So chart out the the first five dealt cards over X number of hands and see if you get a fair distribution of all of the cards.
 
this eliminates the complication of "hold cards" which would affect the number of cards that show up in the draws.It is true each dealt card has a 5/52 chance of being dealt.  It just as true that each drawn card as an n/47 chance of being dealt where "n" is the number of cards being drawn.  So, while it would be more complicated, you could still compare actual and expected in much the same manner.

However, if I were to gaffe a machine with programming, I could make the individual card percentages come out exactly right while still reducing the chances for different paying hands to occur.  For example a larger percentage of ace of clubs when a four-flush of diamonds, hearts, or spades was held could offset a smaller percentage when there was  a four-flush of clubs or the other 3 aces held.

So, outcomes of some type should be involved in any serious test.
[/QUOTE]
I agree with New2VP, Frank - it has to be the outcomes that are analyzed. I heard a story years ago (not sure if it was folklore or true) about how IGT built a VP machine for a Native American casino which tracked the number of RFs that it produced. The casino could request that, for instance, the machine would only allow an RF after 1 million hands were dealt on that machine. The machine would analyze the deal/draw hand, and if it was an RF, compare it to the statistics it had been gathering on that machine. If the 1 million hand threshold had not been reached, it would throw away the deal/draw cards it had selected, and re-draw. This was happening, of course, at incredible speed, imperceptible to the player. The casino was finally required to remove those machines.  
 
This is just the kind of thing that I think people may suspect can happen, and hence would use the tool to help re-assure them of the randomness of the VP games that we play.   

[moneyla]

BillyJoe I would doubt your story about IGT giving the Indian casino a special machine because Nevada Gaming Rules require that any manufacturer selling machines in Nevada must follow Nevada's rules for any machines sold outside of Nevada. This would prevent these "other out of state machines" from finding their way back to Nevada through the resale market or secondary trading.  

[Frank Kneeland]

Once you decide what you
want to test, it is likely that you could either use the binomial
distribution to test for proportions where the observation has only
two outcomes, or a chi-squared test if it has more than two.
Depending on the level of precision required, you may need to choose
a different sample size. Of course, many experiments will be
conducted over a period of time or a number of trials that is based
on the users time and availability and, even with such limitations,
such samples can be analyzed.

I think it pretty much assumed by all
that you are our resident math guru, so I'll be looking to you for
the word on how big the sample needs to be. If it's too large no one
is going to use the utility, so we have to use tricks to get the best
accuracy possible with the smallest sample size. Usability is
paramount.Tests of overall return will be
difficult. Using the normal distribution might require several
hundred thousand hands and other tests would likely need to be
tailor-made from custom distributions.

I agree, a test based on return would
take far too long. Trying to do that is not currently a
goal.I think there is no reason to worry about
being independent of strategy. For most tests, you simply need to
count up the number of trials, number of successes, and be able to
calculate the theoretical probability of a success occurring. A
different strategy simply means a different probability
calculation.

If someone holds a Pay Pair over Two
Pair on a game where optimal strategy says to hold the Two Pair, it's
going to effect the final frequency of 4K when compared to total
hands. I suppose they could use a program like Dan's that calculates
the frequency of each final paying hand based on the strategy people
play, not the optimal strategy. My concern with doing this is I'll
get accused of having an ulterior motive of selling software.
Therefore I must object to anything that would required the purchase
of anything. The utility must be free to all and not require a
software purchase.If you are specific about the
types of events that you will be measuring, the spreadsheet could
have the probability of the event "hard-coded," so this
more difficult portion would NOT have to be completed by the
user.

I think that is best for general use.
The utility MUST be usable by all regardless of their math savvy and
be a matter of simple book keeping.However, if
you do not anticipate exactly what the user would want to test,
perhaps you will also allow "advanced" users to supply
their own probability. For example the probabilities of different
types of quads would be different in Jacks or Better than in Triple
Double Bonus.

I see no problem with including some
user defined functions for the advanced user. But these need to be
optional only.You have conflicting goals of
small sample size and 99% confidence. Higher sample size implies
higher confidence. And as stated earlier, sometimes a larger sample
size is not reasonable to obtain.

Of course I have conflicting goals, I'm
a guy. I've been looking for a nymphomaniac virgin to marry for years
now! Seriously, I need YOU to tell me what is a reasonable level of
certainty to shoot for?Statistical theory (for
frequentists) likes only to reject a given hypothesis or NOT reject
it rather than confirm it with a given percentage, so while you can
reject it with a 99% confidence, you likely will never be able to
confirm a hypothesis with 99% certainty.

I envision a metric on the sheet that
tells the user their level of certainty based on their current sample
size. Certainly that's possible, right?For
example suppose I said that instead of the ace of spades coming up
with probability 1/52, I "knew" from talking to some
programmer that it instead was rounded to 0.019. You would need a
really large sample size to prove one or the other. And if you were
able to, I might say, "Sorry, it was rounded to 4 places; it's
really 0.0192, not 1/52," and put us back to square one (even
worse actually, since we would now require a sample size much bigger
than the previous one).

I understand that incredibly minor
gaffs would be impossible to detect. We can't worry about that or we
are doomed before we start. If we can detect major gaffs or at least
quantify how far within or outside the realm of pure chance people's
results are at least it will be better than nothing at all.


On a science building in Chicago is
chiseled, “When you cannot measure it...your knowledge is of a
meager and unsatisfactory kind”. ~Lord Kelvin

People are currently walking into
casinos, playing as few as 2500 hands and walking out with all sorts
of conclusions based on pure conjecture, anything has got to be
better than that. Anything!I applaud your goal
of having people test this for themselves. There will still be
non-believers, even if they conduct the experiment themselves. There
are even Bayesians among statisticians, who apply weight to prior
beliefs regardless of what an experiment tells them.

Well thank you. As I said in another
thread I'm highly pro skepticism, even if that skepticism is directed
at something I have said or something that I believe. What I believe
strongly is mostly the result of a great deal of personal discovery,
I see no reason why others should be convinced without personal
discovery of their own. For some this is difficult, because a lot of
people slept through statistics class, sure beyond a reasonable
doubt, that they would never use it in their
lifetimes.If you are trying to convince the
inconvincible, this device will still not be "fool"-proof.
Most will have to take any formulas given on faith since they will
not likely be able to derive the formulas themselves. It is
inconceivable for most to understand and therefore believe that it
requires basically the same sample size to estimate a proportion with
a given degree of accuracy (such as the percentage of left-handers
+/- 1%) in a small city of 40,000 as it does in our entire country of
nearly 300 million. And they will likely never be math-savvy enough
to be able to prove that to themselves. They will have to put their
trust in someone who can do this...and of course, how will they know
for certain that the guru did not make a mistake or in fact is
trustworthy.

All the formulas in the utility will be
open source and open for peer review. This meets the requirement for
the scientific method. Anyone that doubts the math is most welcome to
review it themselves or have someone that understands the math review
it for them. Any faults that are found will be published and fixed if
they turn out to be legitimate.
That and the best part is I'm not
trying to convince anyone. I'm trying to build a hammer, what people
choose to bang with it is on them.A purely (or
nearly)
hypothetical example follows: Presume that one person believed that
the proportion of getting the same rank as what was discarded with a
1-card draw was 40% while another believed it was 3/47. Further
presume that a sample of 180 1-card draws was conducted and the
number of matched ranks was 11. The 99% confidence intervals for a
3/47 success probability is 3 to 20; the 99% confidence interval for
the 40% success probability is 55 to 88. Most would clearly see that
the results of the experiment clearly and overwhelmingly favor
randomness, but others might mistakenly feel that an experiment of
several thousand trials would be necessary to get to that
conclusion.

Yes, well how people interpret the data
is not my concern, nor is their final conclusions.

If someone writes me and says I have
used your utility and now believe that “xxxx”. I'm just going to
write them back and say, “Intresting, hope the utility saved you
some time.”A couple other problems:1.
You will from time to time get an extreme sample that suggests that
the population is different than it actually is. Statistics that are
within a given range 99% of the time are outside that range 1% of the
time. When you find a sample in the "critical region" that
is outside the normal range, you will not know for certain if you
have an extreme sample or if there is indeed a particular type of
non-randomness. If you do 100 samples or if 100 people do 1 sample
each, you are expected to get at least one false positive. So there
needs to be room for applying other beliefs in addition to any
experiment. 2. There might be a different interpretation
depending on how extreme the result is. If the chances of getting a
result was 1 in 87 septillion vs. 1 in 101, both would fail a 99%
test, but one would be considerably more convincing than the other.
You may need a bit of a user's manual as there is no substitute for
expert analysis both before and after an experiment is conducted.3.
Frank, I think your inspiration to come up with this idea is terrific
and I wish you well. Please don't interpret what I've written here as
criticism. I hope your project bears fruit and helps some people, but
the debates (fortunately or unfortunately) will continue.

We understand that given 99%
accuracy and 100 people trying it, 1 person would be expected to get
a false positive or false negative. As there is no way to fix this I
recommend not worrying about it. Again something is better than
nothing.
User manual, check!
Criticism? Gee I must have missed
something...All I saw was spot on help with the project, and thanks.
We'll be needing more.

[Frank Kneeland]

Major positive development. I just found a friend that is a visual basic programming expert with skills far beyond anything I've ever seen, that will work for food. OK gourmet comps, but hay it's still food. This will greatly improve the usability and functionality of the final product.Once we lock down what we want coded, he can do it in the blink of an eye. I'm meeting him tonight.~FK

[Frank Kneeland]

I agree with New2VP, Frank - it has to be the outcomes that are analyzed. I heard a story years ago (not sure if it was folklore or true) about how IGT built a VP machine for a Native American casino which tracked the number of RFs that it produced. The casino could request that, for instance, the machine would only allow an RF after 1 million hands were dealt on that machine. The machine would analyze the deal/draw hand, and if it was an RF, compare it to the statistics it had been gathering on that machine. If the 1 million hand threshold had not been reached, it would throw away the deal/draw cards it had selected, and re-draw. This was happening, of course, at incredible speed, imperceptible to the player. The casino was finally required to remove those machines.  
 
This is just the kind of thing that I think people may suspect can happen, and hence would use the tool to help re-assure them of the randomness of the VP games that we play.   What you ask is impossible for most people to accomplish, especially if we track things like the RF. We can include a running made hand tally that will tell you how far within or outside your results are to those predicted by chance alone. That's the best that can be done.This utility isn't going to be a panacea, it's only going to be much better than wild guessing.

[Frank Kneeland]

BillyJoe I would doubt your story about IGT giving the Indian casino a special machine because Nevada Gaming Rules require that any manufacturer selling machines in Nevada must follow Nevada's rules for any machines sold outside of Nevada. This would prevent these "other out of state machines" from finding their way back to Nevada through the resale market or secondary trading.Money, we agreed that we wouldn't question people's claims in this thread. If it's something he's concerned about then it's something we would like to check for if it's possible.There's no shortage of threads that discuss these topics to the point of nausea. We're just trying to make a method/utility/test here. All ideas are welcome, especially if they are hotly debated. Save the right or wrong for another thread...please.You've made some excellent suggestions and helped the thread along, don't ruin your great run now.

[Frank Kneeland]

If was wondering if anyone knows the details about how Nevada Gaming runs that Chi-Squared test?  How many hands do they use?  What condition are they testing? Yes I know a few details, but the way gaming does it requires hooking up to the raw output of the RNG. It's pointless to discuss it as a method for end users.~FK

[Frank Kneeland]

I'll check back in tomorrow. That's all I have time for today.

[New2vp]

I only saw one question that required an answer from me in the long post.  If I missed something else, please re-ask.I agree that the confidence level can be identified based on the sample size used in the test.  Any utility should have that because people would want to use it for their own sample regardless if they have gathered "enough" data.What would also be nice for truer scientists would be the ability to calculate the sample size necessary to obtain a given level of confidence prior to running the experiment.  Then they could decide for themselves how they would resolve the conflict between using a smaller sample size and requiring a higher level of confidence.Once you have a set of finite measures of what you want to test, I can likely help you with such formulae (or confirm what you find elsewhere or suggest modifications).

[moneyla]

[QUOTE=moneyla]

BillyJoe I would doubt your story about IGT giving the Indian casino a special machine because Nevada Gaming Rules require that any manufacturer selling machines in Nevada must follow Nevada's rules for any machines sold outside of Nevada. This would prevent these "other out of state machines" from finding their way back to Nevada through the resale market or secondary trading.Money, we agreed that we wouldn't question people's claims in this thread. If it's something he's concerned about then it's something we would like to check for if it's possible.There's no shortage of threads that discuss these topics to the point of nausea. We're just trying to make a method/utility/test here. All ideas are welcome, especially if they are hotly debated. Save the right or wrong for another thread...please.You've made some excellent suggestions and helped the thread along, don't ruin your great run now. [/QUOTE]
 You're correct Frank, I shouldnt have posted that. Your original rules said: And please note: There should be no need for argument in this thread about whether or not VP machines are fair and random. That would be a conclusion. This thread is not about conclusions, it's about how to formulate them. I guess I was just reacting to the original transgression.