I am looking to build a cluster / grid (super) computer using ~100
standard intel machines as nodes and open source sofware such as Linux
and was looking for examples of performance improvement benchmarks in
simple language in order to sell the idea to my colleagues.

The ideal answer would contain "before" and "after" time taken (in
seconds) on some everyday tasks run in a business environment e.g.,
recalculate a spreadsheet, calculating large regression equation,
running photoshop or some video rendering taskes etc.   (If you read
PC Magazine and PC World benchmarks you know the the type of tasks I
am referring to - althought I would prefer "business" tasks over
"Home/Consumer" tasks). ALso "before" refers to a single processor PC
and "after" refers to the multi processor grid.

ALso I would like to know whether in the "after" state, if there is
any "linearity" (i.e., performace improvement is directly proportionate
to number of PC nodes) or if there is an "exponential" increase (rate of
improvement increases faster than number of PC nodes). Also if there
are examples when the performance improvement "slows down or falls
off" (e.g., no perceptible improvement after the Nth node)

Note: 
In some tasks there might not be any improvement (because these depend
on the network, I/O and disk writing speed rather than processor
speed)

I would love it if the results could compare the time in seconds
(rather than in computer jargon of gigaflops or some other measure.
Its easier to communicate that way)

I DO NOT want information on HOW to build the cluster / grid computer
- only on what performace improvements to expect.

Search strategy
Go to www.ibm.com and go to cluster computing section

http://www-1.ibm.com/servers/eserver/clusters/

Lookup some information here.

Why? 

1.  As someone who has managed a facility with 10 (super?) computer
clusters, I know that the question you are asking is improperly
formulated. and that a business friendly explanation from an
authoratative source such as IBM might be helpful to you.

2.   IBM is in the business of convincing business people to spend
money on computers.  If anyone has business friendly sales material on
grid computing, it would be IBM.  This is based on the idea that their
literature in this area might focus on selling IBM hardware but their
explanation of the business need for the grid cluster "solution" would
be excellent.


Short excerpt from IBM's site.

"One may also characterize clusters by their function:
High Availability (HA):  Redundancy and failover for fault tolerance
High Performance:   Lots of systems working together on a single
problem. A FLOP farm.
Server Consolidation: Central management of resources dedicated to disparate tasks
Initial efforts in Linux clustering have been in the high performance computing "


Given your question, you are only interested in High Performance
business applications.

The single most important "everyday tasks run in a business
environment " for high performance computing is permitting large (in
the thousands) numbers of database transactions to take place. 
Therefore the correct benchmark is # of transactions per time unit for
a given well-defined transaction using a given database
technology/platform.

For computationally intensive tasks such as large scale regression
analysis, it would be variations on FLOPs.

Finally, given that the intent of your question is to locate: 
"examples of performance improvement benchmarks in simple language in
order to sell the idea to my colleagues", you need to specify exactly
the business task for which you wish to improve performance and then
repost the question.  Otherwise, the Google researchers/commentators
are going to find it difficult to provide you with useful information.

As a "business colleague"/"boss who approves expenditure",  I might
find it interesting that there was a performance improvement for some
graphics workshop rendering movie stills for the next special effects
movie, but only a specific example for the exact  business task at
hand would sway me to even consider asking for a few hundred thousand
dollars in the next budget cycle.

Although this comment is not what you were looking for, I hope it may
prove helpful to you.