Configure HTTP Post data input to Nutch before crawling a site

Posted by user365345 on Stack Overflow See other posts from Stack Overflow or by user365345
Published on 2010-06-14T10:37:07Z Indexed on 2010/06/14 10:42 UTC
Read the original article Hit count: 244

Filed under:

nutch

|

crawling

I have to crawl a site which list item based on user input through http post submission. How to configure post http submission details in Nutch.

I got help on how to do HttpPostAuthentication, but I got no help on "how to do post data submit other than username and password".

© Stack Overflow or respective owner

Related posts about nutch

nutch spell checker | nutch navigation filter

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, i am try to configure the nutch 1.0 search engine. First i need to integrate a spell checker or somthing like this, is there a plugin available? My next question is, how to rule out html tag like "", so that navigation is not a part of the index? thanks for all answers >>> More
how to configure nutch on windows and netbeans

as seen on Stack Overflow - Search for 'Stack Overflow'
i have nutch-1.0 and i import it to a web project on netbeans but when i run the project i get a 404 error on the web browser so i want to know : what the wrong ? what are the configurations should i do? note : i want to run nutch for first time thanks >>> More
Problem with running the Nutch command from PHP exec()

as seen on Stack Overflow - Search for 'Stack Overflow'
My Nutch directory lies in /home/myserv/nutch/nutch-1.0/ My php applictaion is in the diretcory /home/myserv/www/ Theres a a php file in my /home/myserv/www/ diretcory that runs a exec command to run a nutch command.PHP code is like : $output = exec("bin/nutch all"); When I run the command… >>> More
Nutch - how to crawl by small patches?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi everyone! I am stuck! Can`t get Nutch to crawl for me by small patches. I start it by bin/nutch crawl command with parameters -depth 7 and -topN 10000. And it never ends. Ends only when my HDD is empty. What i need to do: Start to crawl my seeds with possibility to go further on outlinks. Crawl… >>> More
looking for nutch alternative

as seen on Stack Overflow - Search for 'Stack Overflow'
i am looking for a open source full featured web search engine like nutch , because nutch is complex and it take much time to penetrate its code and i didnt find a book about it . >>> More

Related posts about crawling

https & ajax crawling

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We made on our webpage https://www.1point618.com a transition to ssl and now we using nearly entirely ajax to load the content. Therefore all urls of existing pages have changed. We used the 301 redirect as recommended, also we have implemented google's specification that the webpage is still crawl-able… >>> More
Understanding Ajax crawling of search site

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a couple of questions about Ajax crawling of site, which is kind of search engine itself. The base article explains the mechanism of making AJAX application crawlable. All this stuff with HTML-snapshots is clear and easy to implement, but I cant understand where will Google bot will get "the… >>> More
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm not mad about "not ranking high." I think something is (accidentally) rejecting search engine indexing… >>> More
Ajax site not being crawled - have escaped fragment, what's wrong? [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
My site is anonkun.com. You can see that it's "ajax" and doesn't load much HTML. Here are some example pages: http://anonkun.com http://anonkun.com/?_escaped_fragment_= http://anonkun.com/stories/Dev-kun---FAQ/6ef881f8-cf48-4f87-a688-c585f23809c5 http://anonkun.com/stories/Dev-kun---FAQ/6ef881f8-cf48-4f87-a688-c585f23809c5… >>> More
How to interpret number of URL errors in Google webmaster tools

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out is how to interpret the number of errors over time. At the end of February we've recently migrated… >>> More