This is with ref to stock market figures. Lets say the index
(whichever) has moved up at an average rate (compound) of 10% per
annum. If we have the figures available for the various stocks (daily)
as well as for the index as a whole, how would one check whether the
stocks rise was or was not 'random' (to the extent of tech analysis
not being possible)
How does one rule out the possibility that even though the figures may
seem random, someone may not come up and say they have a 'system'
which works nevertheless. E.g H T T H H T H T T H T H T H is a seres
of heads/tails which seems random enuf but in actual fact there is 1 H
in the first 2 tosses, 2 in the net 4 tosses and 4 in the final 8. A
bit 'TOO random' and therfore not random at all

---

Clarification of Question by myq-ga on 26 Apr 2005 08:16 PDT

One could, for example, take the initial share price figures of ten
years ago and generate the extraploated, periodic (say daily or
monthly)  prices on a basis of 10% p.a.. Then one could test these
figures against the actual share prices at the same periods. But what
sort of tests does one do to find out if the figures meet a randomness
criteria (the philosophical question of whether the figures are
'truly' random may be put aside for now, just whether they seem
reasonably random according to current methods)

---

Clarification of Question by myq-ga on 26 Apr 2005 10:46 PDT

Should one try to fit the data to a poisson distribution, normal dist
or binomial dist, and how should it be done e.g should one calculate
the percentage changes from day to day and expect these to be
distributed normally about the average actual daily percent change?
How would one use a Poisson dist or a bin one? Would it be even make
sense to do this?

---

Clarification of Question by myq-ga on 02 May 2005 10:30 PDT

e.g if one were given the numbers 1, 2, 5, 4, 8, 10, 3, 6, 2, 5, 7, 1
, 3, 8, 4 how would one go about finding out what the chance was that
they were randomly generated from 1 to 10.
A series of all 15  1's would be as likley to obtain from random
generation as the above specific series BUT GIVEN the 1's the chance
of it being random would be a LOT LESS than the chance of the above
series.
How does one calculate the 'LOT LESS'?

The way I read your question: "how do you examine a data set to
determine whether or not there is a hidden relationship between the
data points?"

If you have a computer that's powerful enough, it's almost certain
that you'll find some sort of equation that reasonably fits the
historical data. This equation may have eighty variables and use all
kinds of obscene transformations, but it will fit the historical data.

Will the equation have any sort of predictive value? Not likely. Is
there any rational reason why this equation could have influenced
historical behavior? Not likely. It's just dumb luck. Trying to find
patterns in historical data is basically data mining... you're almost
guaranteed to find a meaningless something.

financeeco-ga , while I too am of the opinion that there are no
patterns in the data I would like to reach this conclusion thorugh
statistical analysis, not simply by saying 'what possibly could cause
a pattern to be really there?'
I am not trying to fit the data to a pattern. I am trying to see
whether the 'theory' that there are patterns even remotely holds up
under rigorous  statistical testing. IMHO it is not enough to simply
state 'what possible reason could there be for patterns' but to
actually find out if these are there or not.
There is nothing mystical about this exercise and many of our opinions
are obtained as a result of such exercises. Is the rate of suicide
higher in the rich? frequently? Do overweight people have a higher
chance of heart disease ? etc etc
If one FIRST poses a hypothesis and THEN tests it the results
(whateever they are) are a lot more likley to be of some significance
than if one simply does data-mining and TRIES to come up with theories
that fit the facts as the facts become known.
If a coin comes up heads 5 times in a row I cannot claim magic after
it comes up; only if I 'pose a hypothesis' BEFORE the tossing that the
coin is going to come up heads 6 times in a row. There is world of a
difference.
It is possible to use historical data as 'future' data by initially
working only on one part of the data and  making a well-defined
hypothesis (which can preferably easily be refuted) and ONLY THEN
seeing if the hypothesis fits the rest of the data.
Statistics can be abused but that does not detract from their very
powerful use in finding out real correlations where the causal
relationships are too complex to be defined by equations.

I may have attempted to answer one of your other questions on google
that fell along a similar vein.

Shannon's information theory will help you, with respect to binary
codes (i.e. H/T, 1's vs 0's).

This should give you more than enough background information to get you where
you want to be:
http://www.cs.utsa.edu/~wagner/laws/coding.html