We've got an indexing application already that crawls several thousand
web sources. But we're constantly looking to add more Sources and I
wonder if there is a Crawler we could "bolt-on" to help us scale up.
We would be looking for an open source tool that could crawl 20-25,000
Sources per day (ie quite a specialist index). But we would constantly
be changing the Sites we wanted it to crawl. Can anyone recommend a
tool - if it works well with Lucene even better as we already use this
for text searches? (potential follow on work for right answer too).