Site crawler/spider that tosses results into mysql

Posted by ian.evans on Server Fault See other posts from Server Fault or by ian.evans
Published on 2010-02-22T15:14:23Z Indexed on 2010/03/17 16:01 UTC
Read the original article Hit count: 575

Filed under:

sphinxsearch

|

mysql

|

crawler

|

spider

It's been suggested that we use mysql for our site's search as it'd be running on the same server that hosts our web server (nginx) and our db (mysql).

Since not all of our pages are created from the database, it's been suggested that we have a crawler that can crawl the site, and toss the page url and data into mysql and have sphinx index on that.

Does anyone know of an open source spider that has a mysql storing option out of the box.

Thanks.

© Server Fault or respective owner

Related posts about sphinxsearch

SphinxSearch or a spider - which one to choose?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, here is my problem: We own SiteA and SiteB and they share the same server and database where we have full control. SiteC , siteD and siteE are some of the sites we own as well but reside on a different web hosts. The goal is to create a unified search functionality for all of the sites mentioned… >>> More
sphinxsearch , Mysql like : order by lang='2' desc , time_popular desc

as seen on Stack Overflow - Search for 'Stack Overflow'
is it possible with sphinx to sort a result like this mysql query? order by lang='2' desc , time_popular desc so get the latest documents with lang=2 in fist position. I really need your help :) >>> More
MySQL error: Can't find symbol '_mysql_plugin_interface_version_' in library

as seen on Server Fault - Search for 'Server Fault'
The boring, necessary details: I'm on Snow Leopard running MySQL locally. I'm trying to install the Sphinx engine for MySQL like so: mysql> install plugin sphinx soname 'sphinx.so'; ERROR 1127 (HY000): Can't find symbol '_mysql_plugin_interface_version_' in library I've Googled everywhere and… >>> More
lucene vs sphinx vs xapian vs lemur , which is a good choice for production use?

as seen on Server Fault - Search for 'Server Fault'
I am evaluating the full-text searching systems and I really need your practical experiences. Also you can suggest some that I didn't mention. >>> More
Pecl complies .so extensions for OSX built-in PHP and not MAMP

as seen on Server Fault - Search for 'Server Fault'
I've installed the sphinx binaries and libraries and am now trying to install the PECL sphinx module. My system is running OS X 10.6 with MAMP 1.8.2 installed. I try to install sphinx using the following command: sudo pecl install sphinx The PECL command outputs the following: running: phpize Configuring… >>> More

Related posts about mysql

How to remove MySQL completely with config and library files on ubuntu 12.04 gnome 3.0

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried everything till now: sudo apt-get remove mysql-server mysql-client mysql-common sudo apt-get purge mysql-server mysql-client mysql-common sudo apt-get autoremove and even more commands... But whenever I am trying to locate mysql. I get a no. of files related to mysql command: shell>>… >>> More
mysql: Cannot load from mysql.proc. The table is probably corrupted

as seen on Server Fault - Search for 'Server Fault'
Mysql was started: /usr/bin/mysqld_safe --datadir=/srv/mysql/myDB --log-error=/srv/mysql/logs/mysqld-myDB.log --pid-file=/srv/mysql/pids/mysqld-myDB.pid --user=mysql --socket=/srv/mysql/sockets/mysql-myDB.sock --port=3700 but when I'm trying to do something: ERROR 1548 (HY000) at line 1: Cannot… >>> More
Why is there a /etc/init.d/mysql file on this Slackware machine? How could it have gotten there?

as seen on Server Fault - Search for 'Server Fault'
A client of my IT-consulting service owns a web-development shop. He's been having problems with a Slackware 12.0 server running MySQL 5.0.67. The machine was set up by the client's sysadmin, who left on bad terms. My client no longer employs a sysadmin. As far as I can tell, the only copy of MySQL… >>> More
mysql: Bind on unix socket: Permission denied

as seen on Server Fault - Search for 'Server Fault'
Can't start mysql with: sudo /usr/bin/mysqld_safe --datadir=/srv/mysql/myDB --log-error=/srv/mysql/logs/mysqld-myDB.log --pid-file=/srv/mysql/pids/mysqld-myDB.pid --user=mysql --socket=/srv/mysql/sockets/mysql-myDB.sock --port=3700 120222 13:40:48 mysqld_safe Starting mysqld daemon with databases… >>> More
MySQL – Learning MySQL Online in 6 Hours – MySQL Fundamentals in 320 Minutes

as seen on SQL Authority - Search for 'SQL Authority'
MySQL is one of the most popular database language and I have been recently working with it a lot. Data have no barrier and every database have their own place. I have been working with MySQL for quite a while and just like SQL Server, I often find lots of people asking me if I have a tutorial which… >>> More