In marketing a new product, how would one calculate the effect of
cannibalization, and when would the effect be negative?

Dana3 --

The American Marketing Association defines a "cannibalization" as the
SALES of a company's new product stealing from the sales of an
existing one:
AMA
"Cannibalization"
http://www.marketingpower.com/live/mg-dictionary-view452.php

"Stealing" sales can be positive or negative, depending on whether or
not profitability increases.  And there are some special cases that
I'll cover at the end.

When Chrysler initially introduced the minivan, it had only one engine
in it: a 4-cylinder engine.  New models "cannibalized" the 4-cylinder
model -- but the company grew sales in all categories to become the
market leader in minivan sales.  Today Chrysler doesn't even produce a
4-cylinder minivan and sales and profits have both increased since the
1983 introduction.

But not in all cases do volumes increase.  So what decision rule does a firm apply?

The rule is to maximize profit -- rather than revenues.  So if a
product has PRODUCT A with a contribution of 60%, sells 1,000 units
and produces a revenue of $1,000,000 -- gross profit is $600,000.

The firm doesn't want to introduce a new model (PRODUCT B) with a 40%
contribution margin and $500 price, if it erodes 300 units from
PRODUCT A  -- even if it adds sales of 800 units for PRODUCT B.  Units
shipped are up; sales are up -- but profit is down:
PRODUCT A: total revenues = 700 x $1,000 = $700,000
PRODUCT B: total revenues = 800 x $500 = $400,000

PRODUCT A: total profit = 700 x $600 = $420,000
PRODUCT B: total profit = 800 x $200 = $160,000

TOTAL PROFIT = $580,000

Dana -- I think that's probably as far as it's intended that you go:
cannibalize revenues; just don't cannibalize total profits.  But the
real world isn't that simple.

---

Now for the exceptions --

First, a company may see a new product emerge which represents a new
category and need to find a way to make a strategic investment.  The
New York Times, the Wall Street Journal, and virtually every other
news source initially had no idea how they'd make money on the
Internet.  News was being posted for free and advertising hadn't
developed; why would a manager invest?  Because it was clear that a
market for electronically-delivered news would emerge.

Here's a fascinating study done of French newspapers, showing the
longer-term impact of the Internet on circulation:
PressFlex
"Cannibalization?  Au Contraire!" (Feb. 8, 2002)
http://www.pressflex.com/news/fullstory.php/aid/35/Cannibalization%3F_Au_contraire!.html

Some other cases where managers will pursue a strategy to cannibalize
their own products are listed in the product strategy presentation
below.  In many cases, the company (or a department like sales) will
not maximize profit over the SHORT RUN because they're worried about
the ability to produce profits in the long run.

For example, Kodak might sell certain products are much lower margin
than normal just to keep Fuji products out of its dealers.  But there
are many reasons to allow cannibalization: preventing competition from
eroding your sales; entering a new market niche; maximizing long-term
profits with a learning curve pricing strategy (which says to price
under cost during early phases of production to maximize diffusion);
and even accounting timing issues.
UAB School of Business
"Product Strategy"
http://bus-web.ad.uab.edu/studentdrive/dayers/MK413/Product%20Strategy.ppt

Best regards,

Omnivorous-GA

Thank you so very much for this answer, it was right on the money! 
Also, thank you for the useful resources.