Is there any descent open-source search engine solutions?

Posted by Nazariy on Super User See other posts from Super User or by Nazariy
Published on 2011-02-11T23:12:18Z Indexed on 2011/02/11 23:27 UTC
Read the original article Hit count: 447

Filed under:

search-engines

|

web-crawler

Few weeks ago my friend asked me how hard is it to launch your own search engine service with list of websites that suppose to be crawled time to time.

First what come at my mind was Google Custom Search however pricing policy is quite tricky and would drain your budget if you reach 500K queries per year.

Another solution I found here was SearchBlox, which can be compared to Google Mini service. It's quite good solution if you planing to cover search over small amount of websites but for larger projects it is not very handy.

I also found few other search platforms like Lucene, Hadoop and Xapian which seems to be quite powerful solutions to reach Google search quality, and Nutch as a web crawler. As most of open-source projects they share same problem, luck of comprehensive guidance of usage, examples and it's expected that you are expert in this subject.

I'm wondering if any of you using this solutions, which of them would you recommend, and what should I be aware of?

© Super User or respective owner

Related posts about search-engines

Your Content Can Get Visibility Through Search Engines - Part 2

as seen on Ezine Articles - Search for 'Ezine Articles'
Your content plays an important role in defining the success of your search engine marketing plan and when you talk about SEO in India, and then you will easily understand different kinds of discoveries and how to use them for your benefit. After you have understood the traditional discovery and some… >>> More
Your Content Can Get Visibility Through Search Engines - Part 1

as seen on Ezine Articles - Search for 'Ezine Articles'
Search engine marketing aims at getting higher search engine rankings for your website but for that you need to get yourself into the search engine Index first. Your content designed for your SEO activity should be accessible to millions of internet users and if your content is not found through the… >>> More
Most Popular Internet Search Engines - Four Tips to Raise Your Website's Search Engine Ranking

as seen on Ezine Articles - Search for 'Ezine Articles'
When you've got a website, one of the most important factors you need to consider is search engine rankings. If you don't have a great ranking position, you won't be easy to find. Many people will only look through the first five pages of search engine results. For that reason, your website must be… >>> More
Tip/Trick: Fix Common SEO Problems Using the URL Rewrite Extension

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Search engine optimization (SEO) is important for any publically facing web-site. A large % of traffic to sites now comes directly from search engines, and improving your site’s search relevancy will lead to more users visiting your site from search engine queries. This can directly or… >>> More
How Does WordPress Blog Search Engines?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, If you go to wordpress admin-settings-privacy, there are two options asking you whether you want to allow your blog to be searched though by seach engines and this option: I would like to block search engines, but allow normal visitors How does wordpress actually block search bots/crawlers… >>> More

Related posts about web-crawler

web crawler needed

as seen on Stack Overflow - Search for 'Stack Overflow'
does anybody know where i can get a free web crawler that actually works with minimal coding by me. ive googled it and can only find really old ones that dont work or openwebspider which doesnt seem to work. ideally id like to store just the web addresses and which links that page contains any suggestions… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
Appengine Apps Vs Google bot web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
i built an appengine web app cricket.hover.in. The web app consists of about 15k url's linked in it, But even after a long time of my launch, no pages are indexed on google. Any base link place on my root site hover.in are being indexed with in minutes. but i placed the same link home page of root… >>> More
Extracting data from internet

as seen on Programmers - Search for 'Programmers'
I would like to extract data from internet like www.mozenda.com does but I want to write my own program to do that. Specific data I'm looking for is various event data. Based on my research, I think custom web crawler is my answer but I Would like to confirm the answer and see if there are any suggestion… >>> More
Web crawler update strategy

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to crawl useful resource (like background picture .. ) from certain websites. It is not a hard job, especially with the help of some wonderful projects like scrapy. The problem here is I not only just want crawl this site ONE TIME. I also want to keep my crawl long running and crawl the updated… >>> More