I'd like information about forecasting the volume of data that will
require cleansing during a migration/conversion effort, specifically,
customer-related data. We're combining customer data from multiple
source systems in order to create one unique record per customer.
We're assuming some of the data won't be consolidated according to our
automated business rules, and want to forecast the volume, and the
staff required to manually clean-up these records.

I'm just looking for some basic stats...doesn't need to be anyting
complex or 100% accurate. I've done some searching myself, but wonder
if anyone else has this information on file, or knows from personal
experience.

1) Approximately 20,000 records will be migrated - how many should we
expect to fail or require manuall cleansing? Any statistics available?
I've heard that you can assume 20% will require cleansing (e.g.,
clean-up of customer names, addresses, combining duplicate customer
records) - is this accurate?

2) How much data will continue to deteriorate each month? Is 3% a fair estimate?

3) I'm wondering how many staff will need to be hired to clean up the
initially migrated data (e.g., 20% of 20 million). Are there any stats
on how long it takes to clean up a record? i.e., 1 minute, 2 minutes,
etc. Or can you recommend a better way to determine the level of
clean-up effort/number of staff required?

---

Clarification of Question by michellita-ga on 10 Apr 2006 14:54 PDT

Thanks for the comment. We will be be looking for patterns in the data
to match up customers as closely as possible, using a combination of
name, date of birth, address, ssn (if available), etc.

Currently there is no key to uniquely identify customers accross
various sources, so are using a clearance/cleansing process to create
a consolidated record with a unique identifier (a customer master
index). We're wondering how many records will require manual
intervention...I know this really does depend on the data, but was
interested in whether any generic statistics were avaiable.

In particular, we're wondering how long and how many people it will
take to clean up the records that cannot be consolidated/matched
through the automated process that is established.

Thanks for your help, and sorry for the delay in responding. I thought
I'd receive an email alert if a response was made, but didn't receive
one.

This is really an open ended question.  It depends on the media, the
formats, etc etc etc.

Do you have a unique key for each customer across data?  
What are your rules going to be?
     How do you identify a unique entity - is Dan Smith at Address A
the same as Address B?
What fields are required
How many fields are there?

There are so many hypotheticals - it really just depends on the data. 
You might find there there is no way to link the records - and that
you have to do them ALL by hand.