To what extent does the conclusion of a statistical test play in the
overall validity of any conclusion drawn about a research study?

The outcome of statistical tests, particularly tests of significance,
plays an important role in measuring the likely validity of a
conclusion arrived at by a research study.  Tests of significance
provide a method for assessing "...the evidence provided by the data
in favor of some statement," (page 461).  As a result, significance
tests appeal to researchers in many fields and are widely used to
report research results.

However, it is a mistake to assume that simply because something is
significant that a researcher's hypothesis is therefore valid or
useful.  A statistically significant effect may be too small to have
any practical significance.  The absence of significance may be in
itself significant and should not be overlooked.  Outliers can cause a
data set to produce highly significant results, presenting a false
impression of the data's predictive value.  Randomness could have
produced the experimental result, particularly if the significance
level is low.  "...[A]ny large set of data -- even several pages of a
table of random digits -- contains some unusual pattern.  Sufficient
computer time will discover that pattern, and when you test
specifically for the pattern that turned up, the result will be
significant.  It also will mean exactly nothing," (page 489).  In
addition, significance does not eliminate the possibility that a
researcher's conclusion of what other than chance caused the
hypothesis to be true is inaccurate.

Because statistical tests are based on the laws of probability,
statistical tests can only state how likely an outcome is.  They
cannot strictly measure the validity of a researcher's conclusions. 
In the absence of a carefully designed experiment, causation can be
difficult to establish.  However, as a good source of evidence,
statistical tests can be an indication of the likelihood of the
correctness of a researcher's conclusions that should be verified by
examining the experiment's design, the researcher's data, and the
explanation's plausibility.

Sincerely,

Wonko

Reference: Introduction to the Practice of Statistics, Moore and
McCabe, W. H. Freeman & Co., 1989