I am looking for a complete list of ".org" domain names on the
internet. The list can be self-made (from DNS servers?) or generated
from another source.
 
The question is answered when a more or less complete list is
delivered in electronic format (I already have 6000 names to check
from).

Thank you in advance,
Kris

---

Request for Question Clarification by pafalafa-ga on 04 May 2003 10:43 PDT

Hello kris4884,

Nice to see you back again...you sure do have a hankering for large
databases, it seems (I answered your "hotels" question a while back).

Anyway, there are 2 million-plus ORG domain names -- I assume you're
looking for a l-o-o-o-n-g list that contains very simple entries like:

www.NAME.org

and nothing else.

Can you confirm, please, that that's what your after.  I'm not sure
how likely it is that a file of all 2 million names can be generated,
but I'm certainly willing to look into it (as are some other
researchers here, I'm sure...).

Thanks.

pafalafa-ga

---

Clarification of Question by kris4884-ga on 04 May 2003 11:02 PDT

Hello again!

I remember indeed...

About your clarification request, indeed, I am looking for the looooooong list.

Just make it the best possible...

Thanks in advance...
Kris

---

Request for Question Clarification by pafalafa-ga on 06 May 2003 13:38 PDT

Just to let you know, I saw your latest comment, and I'm working on
it...as I imagine others are as well.  As soon as one of us can
extract the database that you desire, we'll be sure to let you know.

---

Clarification of Question by kris4884-ga on 06 May 2003 21:40 PDT

OK

---

Request for Question Clarification by pafalafa-ga on 12 May 2003 08:15 PDT

Yes...I'm working on it, and I'm sure other researchers are as well. 
Problem is, getting access to the data itself involves a snail-mail
step and approval of an application by PIR.  The timing is out of our
hands.  When I hear back from PIR, I'll be able to tell you then
whether the task is do-able or not.

Patience....patience....

---

Request for Question Clarification by webadept-ga on 12 May 2003 17:44 PDT

Hi, 

Building the list now. Not going to wait for the PIR everyone else is,
building it from scratch. It is veeerrryyy large :-) Should be done
tonight however. I'll post it as a link to a zipped file, each line an
org domain.

thanks, 

webadept-ga

---

Request for Question Clarification by webadept-ga on 13 May 2003 07:33 PDT

Hi, 

This current list has 112, 738 org domains in text format. The file
can be gotten here:
http://www.lucidmatrix.com/uploads/orgtest.zip

It is not complete, or at least my program has not stopped running,
but I thought after your long wait, you would like something to get
started with, and have some idea of the size of the list you are going
to get.

The way I'm getting this list is using a Perl script to build it, and
a large name list to seek it out. The name list is a combination of
two lists. First is a list of just about all the Proper Names in the
world. At least all of them that can be spelled using my keyboard. It
is about 22, 000 names. The other list is a word list, a dictionary
built using popular novels and writings. With the name list this
builds a 255, 924 word list.

Using each of these words, I'm running the script using the Google
API, and a couple other search engines, using a query like this:

site:.org +yappiness

you can see the results here:
://www.google.com/search?num=100&hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=active&q=site%3A.org+%2Byappiness&btnG=Google+Search

As you can see, only .org sites show up using the site switch, which
is rather nice for this type of search.

After these are gathered, I then dedup the list and put them in
alphabetic order.

The trouble here is that it is taking quite a long time to get through
the whole word list, though I am certain that it will produce a, more
or less, complete list or org sites. I expect the program to be
finished with it's run sometime Friday. If you are okay waiting that
long for the rest of the names, then just post that here and I'll let
the program continue to move forward. If not, keep the list of names
and we'll see if this other source will any faster.

Another thing to note, is that I'm using the "Adult Content" filters
on these searches as well. So the list as it stands will not be
"complete" If you don't want to have this filter, then post that in
your Clarification as well, and I'll change that out. The reason I had
it set is Google and other engines only bring back a maximum of 1000
sites for any given search. To get the maximum "real" org sites, I did
not want the porn orgs taking up those slots.


Thanks, 

webadept-ga

---

Clarification of Question by kris4884-ga on 13 May 2003 10:40 PDT

OK,

you may post the answer when you have the complete result list. 

Can you add in front of the domain names "www" where necessary? Then
the file is ready for immediate use.

Thanks for the help.

For the other researchers, just post a comment when you have the list
and I will post the same question again for the same amount of 100
US$, just for a double check (I really need to have a complete list).

---

Clarification of Question by kris4884-ga on 13 May 2003 21:56 PDT

Hello webadept-ga,

I had some problems working out the list.

Could it be possible for delivery to have multiple text file in the
download zip. Each file should have no more than 100.000 domain names.
The zipped can be downloaded in one time.

About the format of the NAME.org, could you deliver each line in the
text file as following: http://www.Name.org/, where the www should be
there, or replaced by f.ex. another name within the domain.

Thanks a lot for taking in account these comments.

Best regards,
Kris

---

Request for Question Clarification by webadept-ga on 13 May 2003 23:22 PDT

Yes to both questions. I'll adjust all that when the run is done. 

So you want the file broken into smaller files, each with 100 orgs in
them, and then zipped into one download, and the http put in there so
the are more or less full links.

No problem. I'll have this to you as soon as the run is done. 

webadept-ga

---

Request for Question Clarification by pafalafa-ga on 23 May 2003 10:01 PDT

Hello Kris,

Well, after a lot of bureaucratic hassle with the PIR folks, I finally
got access to the ZONE file.  I am now the proud possessor of every
single one of the approx. 3 million goldarn ORG names in existence.

Problem is, this a huge file -- 130 megabytes -- and needs a lot of
clean-up.  It's not an easy task (not for me, anyway) but I'm looking
into ways to make it happen.

But even after the data is massaged into the type of list you want, it
will still be way too large to use in a Windows-type desktop
environment.  This is really material for a larger system than a PC,
probably operating in a UNIX or Linux environment.  Do you have access
to anything like that?

If not, you might want to reconsider just working off Webadept's
initial list of about 100,000 ORG names, since these seem include
pretty much all the major ORG sites.

Anyway...I just wanted to give you an update here.  I'll let you know
if I make any progress.

---

Clarification of Question by kris4884-ga on 24 May 2003 04:36 PDT

Congratulations pafalafa-ga!

I have access to a rather large SUN machine (university stuff), and
some pretty nice WIN2K servers (quadri, 2Gig RAM) so I (or some of my
collegues) should be able to handle the stuff. These machines should
also be able to cleann-up the list too.

If you could split-up the big file in small files of 100K domains each
and put them all together in one .ZIP file for download, then it would
be great.

Just send me a link where a I can download the zip file, and of course
you may answer the question then.

Thank you for your help!
Kris

At long last, here it the list of some 3 million-plus ORG names:

www.binaryheritage.com/ga/orgslist.zip

This will unzip to a single file about 60 megs in size.  I'm sorry I
was not able to break it down into numerous smaller files as you had
asked, but my puny desktop PC just couldn't handle it.  I'm sure it
will be no problem at your end, though.

Let me know if there are any issues or problems with the file at your
end of things...if so, we'll work them out.

I MUST acknowledge the assistance of two of my fellow researchers. 
Without their help, I simply would not have been able to answer this
question.

Seizer-ga, a programming wiz, actually wrote two programs for me to
help me clean up the raw data file and put it in its final form.

Errol-ga provided a link to host the file for, so that you can
download it.

If you comment at all on this answer, I'm sure both Seizer and Errol
would appreciate hearing from you.

Also...Webadept-ga (who was working on your question at the same time
I was...a common occurence here at GA) has offered to compare the file
I've posted with the results of his own search of ORG names.  If he
finds many differences -- possible ORGs that weren't included in my
zone file download from PIR --  I will ask him to post a comment here
to explain what he has found.

This was one of the most challenging questions I've ever
tackled...thanks for giving me that opportunity.

pafalafa-ga

Dear Pafalga-ga,

GREAT, GREAT, GREAT ! Never thought it possible. Many thanks also to
Seizer-ga, Errol-ga,

If you need researchers sticking to it, you just found them.

Sincere thanks!!

Hello kris4884,

You can gain access to the .org Zone files (lists all ORGs) for free,
by filling out the application here :

http://www.orgtransition.info/zone_file

When you do get access, you would simply need to parse the file (Perl
etc..) to get your desired output. I was unable to locate anything
remotely close to the size you are looking for. (There are currently
3.6 million active .org domains ;-)


Please let me know if this route is an acceptable alternative to a
list.


Thanks!
SgtCory

My comment should have stated 2.6 million vice 3.6 million :


.org handover 'successful' by ComputerWire - 28/01/2003
"About 2.6 million names were involved...."
http://212.100.234.54/content/6/29057.html

Thanks for the links, Sarge.  Just so you and Kris know, I've been
checking into the zone files, but I'm reading the question as the
customer wants a clean list delivered as an answer (rather than
getting and parsing the files themselves).

If I've misinterpreted here, I'd appreciate a clarification of the
question, as would other researchers, I'm sure.

I emailed Registrar Services - mailto:sevices@pir.org
and they sent me a reply duplicating the information on
the link that sgtcory-ga posted. It would seem sensible
for all parties to read the Zone File Access Agreement
.pdf file before deciding how to proceed.

Yes, I would like to receive a clear anwser in electronic DB format,
csv is fine. I do not have the knowledge to extract the files myself.

In principle, I 100% agree with the Zone File Access Agreement. It
restricts to use on email, fax or other data. However it does not
restrict the use of the domain names themselves (they cannot be used
for mass mailing). We are an European company, and I do not know how a
such a document is handled, since a signed contract as such in the US
has no validity. So, if a researcher can get me the files, without the
trouble of signing these papers, then I can rize the amount of the
question to 200 US$.

Thanks all for your help.
Kris

Is there any researcher buzzy with this question? Is there a problem
with this? Please give feedback.

Regards,
kris4884

kris4884, and fellow researchers...

For the sake of clarity, this is to inform everyone that
I will not be following up with the list pursuant to the
Zone File Access Agreement via snail-mail. My reason for 
this is simply that, even with access to the full list,
I'm not certain what format it will be in, and am not 
confident in my abilities to reformat the information,
should it be necessary, on the order of 2.6 million entries.

I am perfectly willing to defer to other researchers whom,
I know, have this skill.

Best regards...

sublime1-ga

depending on what format the files come in, i might be able to do it.
lets see how fast they can get me the files.

sorry palfalfa, i didnt notice you were already a lot farther than me.
I'm going to keep working on this out of personal interest, but if
you'd rather post the answer than me, just let me know.

and sorry for butchering your nick, pafalafa

also, the entire database, once cut down to just a list of domain
names, should work fine on an average pc system, as long as it has
plenty of ram (1 gig or more would be ideal) and a windows nt based OS
(win nt, 2000, or xp)