Suppose a six-symbol alphabet is used for encryption, and the relative
frequencies of the six ciphertext symbols (denoted a, b, c, d, e, f)
are 1/4, 1/4, 3/16, 1/8, 1/8, and 1/16, respectively.  Compute the
variance of this system.  Compute the index of coincidence for this
system.

For both questions I use the reference:
http://www.mth.uea.ac.uk/~h320/ciphpdf.pdf .

The variance of a cryptosystem seems to be defined by the sum of the
squares of the differences of the actual frequencies from the
mean frequencies.

In this case, since there are 6 symbols, the mean frequency is 1/6,
and the variance is:

2*(1/4-1/6)^2 + (3/16-1/6)^2 + 2*(1/8 - 1/6 ) ^ 2 + (1/16-1/6)^2

= .02865

There are many references to "index of coincidence", including several
sites that offer index-of-coincidence calculators, such as:
http://www.central.edu/homepages/LintonT/classes/spring01/cryptography/java/trueic.html
and  Most of these definitions are phrased in terms of the index of
coincidence of a specific ciphertext.

At any rate, the definition of index of coincidence in this case is
simply the sum of the squares of the frequencies:

2*(1/4 ^2) + (3/16)^2 + 2*(1/8^2)+1/16^2 =

0.1953

This represents the probability that two randomly chosen characters in
a large ciphertext will be the same.

Search terms:

"index of coincidence" ciphertext

variance ciphertext