Dear jojo27,
Thank you for allowing me to answer the question.
There are 3 major criteria for evaluating measurements ? reliability,
validity and sensitivity.
Reliability
Simply, when the outcome of the measuring process is reproducible, the
measuring instrument is reliable. Reliability applies to a measure
when similar results are obtained over time and across situations.
Broadly defined, reliability is the degree to which measures are free
from error and therefore yield consistent results. For example ordinal
level measures are reliable if they consistently rank order subject in
the same manner ; reliable interval level measures consistently rank
order and maintain the distance between subject.
2 dimensions underlie the concept of reliability i.e. repeatibility
and internal consistency. Assesing the repeatibility of a measure is
the first aspect of a reliability.
Generally there are four methods in confirming reliability:
a)test- retest method
b)split- half method
c)equivalent- form method (alternate form)
d)scorer form method
Test-Retest Reliability
With test-retest reliability a test developer gives the same test to
the same group of test takers on 2 different occasions. Scores on the
1st administration are compared to scores on the 2nd administration
using correlation (r).
This method examines performance over time and gives an estimate of stability.
Often researchers consider test-retest reliability to be a better
measure of temporal stability, which refers to consistency of test
scores, rather than true reliability, which is defined as the ratio of
true to observed variance.
The interval between the administration of the 2 tests can be either a
few hours or several years. Thus, when looking at test-retest
reliability you must be aware of the time interval.
An assumption made with test-retest reliability is that test takers do
not or have not changed over the time period of the 2 administrations.
One concerns of test-retest reliability is termed practice or
carryover effects. Practice or carryover effects are benefits test
takers derive from already having taken a test. This enables them to
solve problems more quickly or correctly the second time they take the
same test.
The reason why practice or carryover effects is of concern as to do
with the attribution of error. Some researchers argue that carryover
effects should be regarded as sources of real stability or instability
in measurement; while others consider it to be a source of measurement
error.
Alternate Forms Reliability (also known as equivalent-form method)
To eliminate practice effects and other problems with the test-retest
method (i.e., reactivity), test developers often give 2 highly similar
forms of the test to the same people at different times. Reliability,
in this case, is again assessed by correlation.
The key aspect of this reliability is to develop an alternate form
that is equivalent in terms of content, response processes, and
statistical characteristics.
Split-Half Reliability
Spilt-half methods of reliability measure the internal consistency of
a test. Remember the measure tape, it has great internal consistency.
The first foot is the same length as the second and third foot, and
the length of every centimeter is also uniform.
Split-half methods also eliminate or reduce the following problems: -
The need for 2 administrations of a test; The difficulty of developing
another form; Carryover and reactivity effects; Changes in a person
over time.
The simplest way to perform spilt-half reliability is to:
1) Administer a test to a group of individuals;
2) Randomly or by some other predetermined method (i.e., split on
similar content; odd-even spilt) divide or split the test into halves
(each half is an alternative form);
3) Correlate the scores on one half with those on the other half.
4) This correlation can be used in estimating the reliability of the test.
A concern with split-half models of internal consistency revolves
around the shortening of a test. When we take a long test, say
100-questions, and spilt it into two 50-questions tests we are
decreasing its reliability. This is because more homogeneous questions
reveal more information about the test takers trait, skill, or
knowledge. This provides more specific information about each test
taker and produces more variation in test scores, which increases
reliability.
For this reason an adjustment the to the split-half reliability is
recommended. The Spearman-Brown Formula can be employed when
estimating the reliability using the split half method.
rxx = k r / (1 + (k ? 1))r
Where: k = number of items in the 'new' split-half test (i.e., usually
the original number of questions your test had before you split it)
divided by the number of original items in the split-half test (i.e.,
the number of question in your split-half correlation). In other
words, the number of times longer the 'new' test will be. For
example, assume your test has 80 questions. Your perform a spilt-half
reliability and obtain an r = 0.8. That r = 0.8 is based on 40 items.
Those 40 question comprise your original items in the split-half
test. Now you want to adjust your reliability since the test actually
had 80 questions. 80 is considered the length of the 'new' test.
Thus, k = 80/40 = 2. Note: The Spearman-Brown formula is also used to
estimate how much a test's reliability will increase when the test is
increased by adding parallel items.
r = the correlation between the original split-halves.
Another weakness in the spilt-half method is the number of different
ways a test can be spilt.
Some splits may give a much higher correlation than other splits.
An even better way to measure internal consistency is to compare
individuals' scores on all possible ways of splitting the test in
halves. This will compensate for any error introduced by any lack of
equivalence in the two halves. We can do this by describing the
amount of intercorrelation between questions on a test or subscale,
and the number of questions we have on the test.
1) KR-20 (Kuder & Richardson, 1937, 1939). This method is used for
tests whose questions can be scores as either 0 or 1.
2) Coefficicent alpha (Cronbach, 1951). This method is used for
questions such as rating scales that have 2 or more possible answers.
Another problem for the split-half method is whether the test being
split is homogeneous (i.e., measuring one characteristic) or
heterogeneous (measuring many characteristics).
One solution here is to determine reliability for each heterogeneous
component of the test and then compare, using correlation, those
components.
Note: Your text makes the distinction between Split-half methods of
reliability and internal consistency methods. Many researchers and
most other texts consider split-half reliability as a component of
internal consistency.
Scorer Reliability
The above methods have not considered the person who administers the
test? Can individuals make mistakes in scoring that add error? Of
course!
Thus, judgements or ratings made by different scorers are often
compared using correlation to see how much they agree.
This is termed the scorer reliability or the inter-rater reliability.
Search strategy:
Statistics, reliability, test retest method |
