Hello again Hullcapital-ga,
Let me say at the outset that you have bitten off quite a mouthful
with your interest in the nuts and bolts of the CPI. Entire
government bureaus, academic departments and partisan think-tanks have
tried to answer your question of whether the techniques used to
calculate CPI introduce unreasonable bias into the numbers.
No one has yet arrived at a definitive conclusion. I have given you
some of the key sources of information below. I believe these are
what you are looking for, and will help you understand the context of
how CPI calculations have changed over the years.
IF FOR ANY REASON this answer does not meet your needs, don't despair.
Just let me know with a Request for Clarification what additional
information you need. I will be glad to accomodate your request as
best I can.
----------
The Consumer Price Index is calculated by the U.S. Bureau of Labor
Statistics within the Department of Labor. The methods use to
calculate CPI are continually changing in response to changing
consumer habits, new items appearing on the market (and old ones
disappearing) and responses to both crticism of the survey and
computation methods, as well as the need to respond to budget
realities and pare costs of conducting the survey.
In short, there have been hundreds, if not thousands, of changes made
to the CPI in a continuing process.
There are many documents that discuss the changing methods used to
calculate the CPI, and I will reference these towards the end of this
answer. But the motherlode of information on the history of changes
to the CPI is the manual used by the federal government itself. The
methodologies for CPI calculation, and the changes that have occured
in these methods, are fairly well-documented in Chapter 17 of the BLS
Handbook of Methods:
http://stats.bls.gov/opub/hom/homch17_itc.htm
U.S. Department of Labor
Bureau of Labor Statistics
BLS Handbook of Methods
Chapter 17.
Consumer Price Indexes
The chapter is divided into three main parts:
Part I. The Index in Brief
Background
Concepts
Scope and Calculation
Uses
Analysis and Presentation
Limitations of the Index
Part II. Construction of the Index
Index Calculation
Annual and Semiannual Average Index Estimation
Sampling: Areas, Items, and Outlets
Item and Outlet Samples
Estimation of Price Change
Average Prices
Part III. Precision of Estimates
Estimating Variances of the Index and Price Change
Estimating Variances Starting in 1987
Variance Estimates for the Index
Variance Estimates for Price Change
Nonsampling Error
Technical References
I've listed these in detail so you'll have an idea of the content of
the chapter, beyond what I summarize and excerpt here. All text below
is from the manual itself, except for comments that I've added, which
are in brackets.
[Let's start with the introduction to the Chapter, which is worth
excerpting at length, as it gives a good context for the various
meanings of CPI]
The Consumer Price Index (CPI) measures the average
change in the prices paid by urban consumers for
a fixed market basket of goods and services. The Bureau
of Labor Statistics (BLS) calculates the CPI monthly
and publishes it about 2 weeks after the end of the month to
which it refers.
BLS calculates the CPI for two population groups, one
consisting only of wage earners and clerical workers and the
other consisting of all urban consumers.1 The Consumer Price
Index for Urban Wage Earners and Clerical Workers (CPIW)
is a continuation of the historical index that was introduced
well over a half-century ago for use in wage negotiations.
As new uses were developed for the CPI in recent years,
the need for a broader and more representative index became
apparent. The Consumer Price Index for All Urban
Consumers (CPI-U) introduced in 1978 is representative of
the buying habits of about 80 percent of the non-institutional
population of the United States, compared with 32 percent
represented in the CPI-W. The methodology for producing
the index is the same for both populations and is described
in detail in part II of this chapter.
[The chapter goes on to give some of the history of the CPI, dating
back World War I. Since your interest is in changes of the past 20
years or so, I'll skip the history, but it's there if you want to have
a look. A substantial change to data methods was made in 1978, which
is where I'll begin.]
The 1978 revision reflected spending patterns based upon
the surveys of consumer expenditures conducted in 1972-
74. A new and expanded 85-area sample was selected based
on the 1970 Census of Population. The Point-of Purchase
Survey (POPS) was also introduced. POPS eliminated reliance
on outdated secondary sources for screening samples
of establishments or outlets where prices are collected. A
new store-specific approach to the item selection process was
also introduced, as was a second index, the more broadly
based CPI for All Urban Consumers. The CPI-U took into
account the buying patterns of professional and salaried
workers, part-time workers, the self-employed, the unemployed,
and retired people, in addition to wage earners and
clerical workers.
In January 1983, the Bureau changed the way in which
homeownership costs were measured.7 A rental equivalence
method replaced the asset-price approach to home-ownership
costs for the CPI-U. In January 1985, the same change
was made in the CPI-W. The central purpose of the change
was to separate shelter costs from the investment component
of homeownership so that the index would reflect only
the cost of shelter services provided by owner-occupied
homes.
The most recent revision of the CPI, completed in 1987,
further improved sampling, data collection, processing, and
statistical estimation. This revision stressed techniques that
would make the production and calculation of the CPI more
efficient, especially with respect to design and allocation of
the samples. The updated samples of items, outlets, and areas
were based upon data from the Survey of Consumer Expenditures
for the years 1982, 1983, and 1984; the 1980
Census of Population; and the ongoing Point-of-Purchase
Survey, which, beginning with 1985, reflected the new item
and area design. The new technique of rolling in the new
area, item, and outlet samples significantly reduced the cost
of introducing new samples. In addition, the housing survey
was redesigned to represent optimally both owners and renters,
which improved the estimation method for shelter costs.
[The Chapter also notes that additional changes are due to go into
effect in 1998, obviously dating the text itself. Post-1998 changes
can be found in other sources, discussed below. A summary of the
major changes made to the CPI from 1966-1996 can be found in the BLS
table at:
http://stats.bls.gov/opub/hom/homch17_b_t1.htm
[This may be your single best source of summary information on recent
changes of signficance].
[An important point about CPI is that the goods are "weighted"
according to total consumer expenditures on the good, as well as on
its "relative importance", as discussed here]
Weights and relative importance
The weight of an item in the CPI is derived from the
expenditure on that item as estimated by the Consumer Expenditure
Survey. This survey provides data on the average
expenditure on selected items, such as white bread, gasoline,
and so on, that were purchased by the index population
during the survey period. In a fixed-weight index such
as the CPI, the implicit quantity of any item used in calculating
the index remains the same from month to month.
A related concept is the relative importance of an item.
The relative importance shows the share of total expenditure
that would occur if quantities consumed were unaffected
by changes in relative prices and actually remained
constant. Although the implicit quantity weights remain
fixed, the relative importance changes over time, reflecting
the effect of price changes. Items registering a greater than
average price increase (or smaller decrease) become relatively
more important. Conversely, items registering a
smaller than average price increase (or larger decrease) become
relatively less important. Thus, the relative importance
of medical care in the index for all urban consumers,
which was 6.0 percent in December 1982, was 6.9 percent
in December 1986. During the same period, the relative
importance of energy fell from 12.4 percent to 8.9 percent.
The published data on relative importance are often used to
answer such questions as: What was the direct effect on the
overall CPI of a particular price change (e.g., gasoline prices)
for a particular period? (See appendix 2.)
[The actual summary description of ow the CPI is compiled is finally
discussed]
Scope and Calculation
Prices for the goods and services used to calculate the
index are collected in 85 urban areas throughout the country
and from about 21,000 retail and service establishmentssupermarkets,
department stores, gasoline stations, hospitals,
etc. In addition, data on rents are collected from about
40,000 landlords or tenants and 20,000 owner occupants
are asked about their housing units.
Food, fuels, and a few other items are priced monthly in
all 85 locations. Prices of most other goods and services are
collected monthly in the five largest urban areas and bimonthly
in the remaining areas. Trained BLS field representatives
collect all price information through visits or telephone
calls.
The index is the average of the price changes for all the
various items within each area. The averaging process uses
weights which represent the importance of the items in the
spending pattern of the appropriate population group in that
area. Separate indexes are compiled for: the urban United
States, 4 regions, 4 size classes, 13 groups cross-classified
by region and population size, and 29 local metropolitan
statistical areas.
[The BLS explicitly recognizes limits in the CPI]
Limitations of the Index
CPI users should understand that the CPI may not be
applicable to all questions about price movements for all
population groups. For example, the indexes represent the
average movement of prices for the U.S. urban population
and, thus, may not be appropriate for use by non-urban residents.
Also, the CPI does not provide data separately for
the rate of inflation experienced by any demographic subgroup
of the population, such as the elderly, which may
purchase different consumer items and face different rates
of inflation. And, as mentioned earlier, the CPI is an index
of price change, not a cost-of-living index.
In addition, the indexes cannot be used to determine relative
living costs. The change in an index for individual geographic
area index measures how much prices have changed
in that particular area. It does not show whether prices or
living costs are higher or lower in that area relative to another.
A further limitation is that the CPI is not a complete
measure of price change. Because the index is estimated
from a sample of consumer purchases, the results may deviate
slightly from those which would be obtained if all consumer
transactions were covered. These estimating or sampling
errors are statistical limitations of the index.
-------------------------------------------
[Parts II and III of the manual are very specific, very detailed, and
steeped in formulas and arcane sampling techniques...if you really
want to dig into how the CPI has changed over the years, this is
probably where you have to do a lot of deep digging through text that
looks like this:]
From 1979 through 1984, unclustered samples of households
were selected for the survey. Since 1985, households
have been selected on the basis of a noncompact clustering
procedure which is modeled after the sample design for the
Consumer Expenditure Survey.
[You get the idea...these are potentially significant changes, and are
strewn throughout the chapter, but it would take a PhD thesis to
really uncover the impact...if any...on the overall CPI data].
[Another constant source of change is the changing nature of consumer
goods. The manual discusses this at length, and begins as follows:]
Item substitution, quality adjustments, and
imputation
One of the more difficult conceptual problems faced in
compiling a price index is the accurate measurement and
treatment of quality change due to changing product specifications
and consumption patterns. The concept of the CPI
requires that BLS measures through time, the cost of purchasing
a fixed, constant-quality market basket of goods
and services. In reality, products frequently disappear, products
are replaced with new versions, and new products
emerge. BLS uses several methods to adjust for quality
change and to account for the change in item specifications.
These methods may be categorized as 1) directly comparable,
2) direct quality adjustment, and 3) imputation. In
all cases, it is necessary to estimate a new base-period price
in order to use the new item specification in future periods.
[Changes in the practice of medicine can be particularly confounding]
Medical care. Another area in which quality adjustment presents
particular difficulties is medical care. Not all factors
affecting the quality of medical care services can be accounted
for in the description of the item being priced. Quite
often the respondent does not have knowledge of many pricedetermining
quality factors. For example, hospital room
modifications, changes in the nurse-to-patient ratio, or the
availability of new equipment are all likely to contribute to
determining the price level of the room service priced. Such
changes are normally reflected as price movement because
BLS either is not aware of the changes or has no method
available to deal with the change. Improved technologies
and procedures can lead to quality changes that cannot necessarily
be measured by BLS.
-----------
That's it for the BLS Manual.
The bottom line on all this is probably something like this: the
biggest impact in measuring the CPI from year to year probably stems
from changing conditions in the market place -- new goods coming in,
old goods disappearing, changes in consumer tastes, changes in habits
(like the emergence of manufacturer's rebates as a selling technique)
and so on.
These changes make year to year comparisons difficult, at best, and
over the longer term, comparisions become extrememly problemmatic.
The BLS tries to adjust its techniques to deal with the world as they
find it, and to have reasonably comparable data from year to year. My
gut reaction is that the monkeywrenches thrown into the system
probably have more to do with changing market conditions, that with
the actual techniques used to sample, weight, and ultimately compute
the CPI.
----------
As I mentioned earlier, there is a great deal of additional
information available on the CPI that discusses both current practices
as well as historical changes to the system.
Some of this is available on the BLS CPI page directly:
http://www.bls.gov/cpi/
and includes:
Table Containing History of CPI-U U.S. All Items Indexes and Annual
Percent Changes From 1913 to Present
Relative Importance of Components in the Consumer Price Index December
2001:
--US City Average using 1993-95 weights
--US City Average using 1999-2000 weights
[NOTE: These contain some of the post-1998 changes that I mentioned
above]
[To see the impact of a specific set of revisions on the CPI, you can
have a look at the reports and links here:]
Values for some series--both CPI for All Urban Consumers (CPI-U) and
CPI for Urban Wage Earners and Clerical Workers (CPI-W)--have been
revised. The revisions cover the period from January 2000 through
August 2000. These revised values were loaded into the LABSTAT
database at 10 a.m. EDT on Thursday, September 28, 2000. See
http://www.bls.gov/cpi/cpirev01.htm for more information, and for
links to tables containing the revised index values.
[A whole series of component-by-component facts sheets are available,
along with some gneeral information]
Understanding the Consumer Price Index: Answers to Some Questions (PDF
15K)
Guide to Available CPI Data (PDF 15K)
How to Obtain CPI Information
How BLS Measures Changes in Consumer Prices
How to Use the Consumer Price Index for Escalation
Measuring Price Change for Medical Care in the CPI
The Consumer Price Index--Why the Published Averages Don't Always
Match An Individual's Inflation Experience
Consumer Price Indexes for Rent and Rental Equivalence
How BLS Measures Price Change in the Consumer Price Index for Cable
Television
How BLS Measures Price Change for Lodging Away from Home in the
Consumer Price Index
How BLS Measures Price Change For Household Fuels in the Consumer
Price Index
How BLS Measures Price Change for Cellular Telephone Services in the
Consumer Price Index
How BLS Measures Price Change For Local Telephone Services in the
Consumer Price Index
How BLS Measures Price Change For Long Distance Telephone Services in
the Consumer Price Index
How BLS Measures Price Change for College Tuition and Fees in the
Consumer Price Index
How BLS Measures Price Change For Elementary and High School Tuition
and Fees in the Consumer Price Index
How BLS Measures Price Change For Airline Fares in the Consumer Price
Index
----------
Finally, and in case you don't have enough reading material yet, there
is (at least) one more document worth having a look at. The Federal
Reserve Board of San Francisco put out a very concise discussion of
possible bias in the CPI:
http://www.frbsf.org/econrsrch/wklyltr/el97-16.html
Bias in the CPI: "Roughly Right or Precisely Wrong"
where they talk about how the CPI is pieced together, and what biases
are introduced by some of external (market place changes) and internal
(sampling and modelling techniques) that go into making CPI
determinations.
-----------
I hope this meets your needs, but like I said earlier, just give me a
shout if I can do anything else to assist you here. |
