Google Answers Logo
View Question
 
Q: Automatic Spider/Crawler of news websites ( No Answer,   2 Comments )
Question  
Subject: Automatic Spider/Crawler of news websites
Category: Computers
Asked by: corporeo-ga
List Price: $30.00
Posted: 15 Jan 2006 10:16 PST
Expires: 14 Feb 2006 10:16 PST
Question ID: 433690
Where can I find and download software that will do the following:

1) Extract entire news websites (10 to 15 Mexican news websites) and
save them to hard drive keeping the files relative structure
(hopefully it will spider all types of web page formats and media)
2) The extraction has to be automatic and daily (therefore has to be
able to assign different folder names to different days)
3) Reviewing the saved information can be done in a normal browser
(and NOT an internal browser)

My objective (I think it would help if I explain) is to study the
presidential campaign in Mexico by saving the following 6 months of
online news. After the information has been collected it will be
analyzed with content analysis packages.
I know there are many spiders and crawlers out there. It is the
automated daily collection of information in different folders that
matters.
Answer  
There is no answer at this time.

Comments  
Subject: Re: Automatic Spider/Crawler of news websites
From: eliteskillsdotcom-ga on 15 Jan 2006 15:03 PST
 
Im pretty sure Iopus can do it.

However, if you're experienced with computers you may find it worth
the money to download a program like wget, httrack, or free download
manager. I forget which but some have synchronization features you can
use. Wget is good because even if it doesnt have synchronization built
in, it's command line based so you can download with a cron job or
windows schedueler.




Iopus: http://www.iopus.com/iim/
WGET: http://www.gnu.org/software/wget/wget.html
HTTrack: http://www.httrack.com/
Free Download Manager: http://www.freedownloadmanager.org/
Subject: Re: Automatic Spider/Crawler of news websites
From: padpub-ga on 16 Jan 2006 10:04 PST
 
Try NewzCrawler software. I think it may meet your requirements. Visit
following pages for more information:

http://www.newzcrawler.com/
http://www.newzcrawler.com/features.shtml

regards,
padpub
http://www.clicktry.com/

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy