Nutch - how to crawl by small patches?

Posted by Yurish on Stack Overflow See other posts from Stack Overflow or by Yurish
Published on 2010-03-29T12:40:01Z Indexed on 2010/03/29 12:43 UTC
Read the original article Hit count: 430

Filed under:

nutch

|

lucene

|

web-crawler

Hi everyone!

I am stuck! Can`t get Nutch to crawl for me by small patches. I start it by bin/nutch crawl command with parameters -depth 7 and -topN 10000. And it never ends. Ends only when my HDD is empty. What i need to do:

Start to crawl my seeds with possibility to go further on outlinks.
Crawl 20000 pages, then index them.
Crawl another 20000 pages, index them and merge with first index.
Loop step 3 n times.

Tried also with scripts found in wiki, but all scripts i found don't go further. If i run them again, they do everything from beginning. And in the end of script i have the same index i had, when started to crawl. But, i need to continue my crawl.

Some help would be very usefull!

© Stack Overflow or respective owner

Related posts about nutch

nutch spell checker | nutch navigation filter

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, i am try to configure the nutch 1.0 search engine. First i need to integrate a spell checker or somthing like this, is there a plugin available? My next question is, how to rule out html tag like "", so that navigation is not a part of the index? thanks for all answers >>> More
how to configure nutch on windows and netbeans

as seen on Stack Overflow - Search for 'Stack Overflow'
i have nutch-1.0 and i import it to a web project on netbeans but when i run the project i get a 404 error on the web browser so i want to know : what the wrong ? what are the configurations should i do? note : i want to run nutch for first time thanks >>> More
Problem with running the Nutch command from PHP exec()

as seen on Stack Overflow - Search for 'Stack Overflow'
My Nutch directory lies in /home/myserv/nutch/nutch-1.0/ My php applictaion is in the diretcory /home/myserv/www/ Theres a a php file in my /home/myserv/www/ diretcory that runs a exec command to run a nutch command.PHP code is like : $output = exec("bin/nutch all"); When I run the command… >>> More
Nutch - how to crawl by small patches?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi everyone! I am stuck! Can`t get Nutch to crawl for me by small patches. I start it by bin/nutch crawl command with parameters -depth 7 and -topN 10000. And it never ends. Ends only when my HDD is empty. What i need to do: Start to crawl my seeds with possibility to go further on outlinks. Crawl… >>> More
looking for nutch alternative

as seen on Stack Overflow - Search for 'Stack Overflow'
i am looking for a open source full featured web search engine like nutch , because nutch is complex and it take much time to penetrate its code and i didnt find a book about it . >>> More

Related posts about lucene

performance comparision between Zend Lucene and Java Lucene

as seen on Stack Overflow - Search for 'Stack Overflow'
Zend Lucene and Java Lucene are built in PHP and java repectively, and PHP language has a higher level than java. Just wondering How big the performance difference among these two, regarding to index building and data searching? Is it much more effective to let java create and rebuild index, and… >>> More
Why wasn't fast-vector-highlighter (lucene-contrib) made an official part of Lucene 3.0 core

as seen on Stack Overflow - Search for 'Stack Overflow'
I've read some Jira entries and they mentioned moving fast-vector-highlighter to core about a year ago but it never made it. Looking at the svn for contrib it seems incomplete. There are no tests for FastVectorHighlighter Documentation is lacking No samples anywhere on apache.org Anyone have… >>> More
pylucene: install error

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to install Pylucene (pylucene-3.3-3-src.tar.gz) on my ubuntu linux 11.10. I have python 2.7.2. I was able to compile JCC (I think) because I didnt see any error when I installed it. When I tried to install Pylucene I get the following error. Can someone help? Thanks. ICU not installed /usr/bin/python… >>> More
Solr WordDelimiterFilter + Lucene Highlighter

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to get the Highlighter class from Lucene to work properly with tokens coming from Solr's WordDelimiterFilter. It works 90% of the time, but if the matching text contains a ',' such as "1,500" the output is incorrect: Expected: 'test 1,500 this' Observed: 'test 11,500 this' I… >>> More
java AbstractMethodError

as seen on Stack Overflow - Search for 'Stack Overflow'
How to handle this error in lucene: java.lang.AbstractMethodError: org.apache.lucene.store.Directory.listAll()[Ljava/lang/String; at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:568) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69) … >>> More