We're LearnOutLoud.com and we're going to be selling audio books real
quick here. We're working with the Ingram Book Company and we've
received their massive 400 MB stock@ingram.dat file. It contains over
1.5 million rows so we can't open it up with Excel. It's too large.
I've opened it up with notepad and it takes about 15 minutes to open.
Our programmer has written a script so that when we save it as a CSV
file then we will pull the stock info and pricing for the matching
ISBNs we have on our site to determine if we're selling them or not. I
can't save it from Notepad as a CSV file because it says there's not
enough memory to perform that operation. We don't really want to
convert the whole file to CSV and upload the 400 MB file back to our
server. What I was able to do was grab 50,000 rows at a time from the
notepad file and copy them into an Excel sheet and make them
delineated according to spaces. This way I can sort according to
format and find the formats of audio CD and cassette which is all we
currently want, and then I can grab those rows, put them in notepad,
and save that as a CSV file. That worked, but the only problem is I'd
have to do this 30 times with Excel since Excel won't allow for over
50,000 rows and 1,500,000/50,000 = 30 times. Are there any Excel-like
programs that will put data in rows and columns that you can sort, but
that will allow for 1.5 million rows or at least many more rows than
50,000? I've heard this might be possible with Microsoft Access. How
do I handle such massive amounts of data?! Thanks.

Hi jpbischke,

Let's try with Ms Access 2000 tool. Follow the instructions:
1. Rename the stock@ingram.dat into stock@ingram.txt
2. Create a Ms Access file (e.g. db1.mdb)
3. Save it to C:\temp\db1.mdb
4. Highlight or click TABLE object
5. Click FILE menu then choose GET EXTERNAL DATA then click IMPORT 
6. Browse to the folder of stock@ingram.dat (e.g. C:\temp\stock@ingram.txt)
7. Choose "Text Files" from Files of types drop down button
8. Highlight stock@ingram.txt and then click IMPORT button
9. Determine the the separator of the column. Choose Delimited or
Fixed Width. If necessary click ADVANCE button to define the data
type.
10. Click the NEXT button and follow the instruction.

Well, hope this help you. Should you have any questions please feel free to ask.

Thx,
DenbagusID

The problem I'm having is that I can't save it as a txt file. I think
Access will work. It definitely allows for a lot more rows. But when I
open stock@ingram.dat in a notepad and try to save it as
stock@ingram.txt it always says not enough storage or something like
that. And it's not hard drive space. I think it's running out of
virtual memory or something. And I can't open the DAT file with
Access. I can open a text file with Access and delineate it and sort
according to how I want to so that's the solution. I just need to
figure out how to make this stock@ingram.dat into stock@ingram.txt.

Without having a great knowledge of "DAT" filetype, is it not possible
to simply rename the file as stock@ingram.txt (rather than .dot) on
your PC, and then open it in access as detailed previously?

Any processing of this type data screams for a simple stream-processed solution.
I would suggest writing a short Perl script that will process a line
of the original stock@ingram.dat file looking for the data you need.
For instance, if you have a list of ISBNs that you want details on
then you would with your script

1. Have the script read the ISBNs you are interested in.
2. Process stock@ingram.dat a line at a time, extracting relevant data
and writing this an output file (say a CSV)

Then you can read the CSV into Excel or Access or whatever is your
favourite database.

txt
DAT
csv

they are all text files bud, and theirs no point converting between
them if you have a dat file import it into ms-access directly, just
like DenbagusID directed. If your running out of virtual memory go to

Control Panel>System>Advanced>Performance.

Click settings then advanced.

It will allow you to change your page size manually if really needed. 

But please remember, dat, csv, txt its all just text.