A question for those trained in mathematical statistics.

Let's suppose I want to create a rating system for researchers on
Google Answers along the following lines. Let's say that each of the
question askers rate the answers they receive to their questions as
either 1 (positive), 0 (neutral), -1 (negative). If a given researcher
has had X positive responses, Y negative responses and Z neutral
responses after answering N questions, let's define the researcher's
"score" (S) as S = (X-Y)/N.  Let's define a researcher's "true score",
T, as the limit of that researcher's score as the number of questions
that researcher answers gets forever larger (assuming a
limit exists).

Now let's call a researcher "reliable" if the researcher's true score,
T >= R where R is a constant between -1 and 1.

If a researcher's score after N answers is S, what is the probability
that that researcher is reliable?   (I'm interested in the cases for
both small and large N)

Please answer as specifically as possible i.e. an algorithm or formula
for calculating is the ideal answer here. Intelligent suggestions for
improvements on the reliability system calculation proposed above are
also welcomed but they are only of secondary importance.

Dear mrfickle,

I need some clarifications regarding your questions. By the way, why
are you making this question so personal ? It may get against the
rules as stated in #34 in researcher guidelines ;)

Coming to the point, you are not so clear about the variable R. I
guess R is a degree of reliability, and if a researcher scores T >=R,
he/she should be considered reliable at that scale. Right ?

After that, you shall need the value of P(S>R). Are you assuming S to
be a random variable ? That requires knowledge of the probabilities
with which X,Y,Z responses are acquired by the researcher. And if S is
a numerical value, I don't really get your question, as it boils down
to a deterministic case.

Regards,
vsingh.

Firstly, yes you are right about the definition of R. R is a constant
chosen in advance, not a random variable.

My statistics is somewhat rusty (hence the reason I can't solve this
myself!) so forgive me if my terminology is incorrect. The reason why
this is not a deterministic problem is that S is an estimator of T. As
N becomes larger, S tends towards T but there is a degree of
randomness associated with this estimate for which an appropriate
distribution is required to estimate that randomness. I can assure you
this is a statistics problem, not a "deterministic" one.

By the way, I am happy to change the context of the question to
involve Beetlejuice researchers if it makes you more comfortable :)

Dear Questioner,
                  Not much more than guessing but given similar or
related questions have come up before and having seen the responses to
these I would suspect that although your question may have had some
relevance and certainly looked interesting, it was of the type frowned
upon by GOOGLE ANSWERS administrators which  most researchers would be
aware of and hence reluctant to get into, especially as a formalised
fee paid question