Best way to store data for Greasemonkey based crawler?

Posted by Björn on Stack Overflow See other posts from Stack Overflow or by Björn
Published on 2009-01-28T14:23:05Z Indexed on 2010/05/16 13:00 UTC
Read the original article Hit count: 263

Filed under:

greasemonkey

|

crawler

|

persistence

|

storage

|

xmlhttprequest

I want to crawl a site with Greasemonkey and wonder if there is a better way to temporarily store values than with GM_setValue.

What I want to do is crawl my contacts in a social network and extract the Twitter URLs from their profile pages.

My current plan is to open each profile in it's own tab, so that it looks more like a normal browsing person (ie css, scrits and images will be loaded by the browser). Then store the Twitter URL with GM_setValue. Once all profile pages have been crawled, create a page using the stored values.

I am not so happy with the storage option, though. Maybe there is a better way?

I have considered inserting the user profiles into the current page so that I could all process them with the same script instance, but I am not sure if XMLHttpRequest looks indistignuishable from normal user initiated requests.

© Stack Overflow or respective owner

Related posts about greasemonkey

Greasemonkey @require jQuery not working "Component not available"

as seen on Stack Overflow - Search for 'Stack Overflow'
I've seen the other question on here about loading jQuery in a Greasemonkey. Having tried that method, with this require statement inside my ==UserScript== tags: // @require http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js I still get the following error message in Firefox's error… >>> More
Firefox Password script same PW auto login for 1000s devices Possible to use greasemonkey

as seen on Stack Overflow - Search for 'Stack Overflow'
I currently work in a position that i have to manage and access 1000s of pages for troubleshooting and new setup.... and im trying to figure out a way for firefox or chrome to setup ANYtime it sees for instance a a web based page for equipment like (CISCO, Linksys, Sonicwall, T1 controllers) from… >>> More
.submit() doesn't work with Firefox and Greasemonkey...

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to make an auto login script and I'm stuck on the submit part... The source of the submit form from the website is <input type="submit" value="Sign In" class="signin" id="sumbitLogin"> and I'm trying document.getElementById("sumbitLogin").submit(); if I set an Attribute, for… >>> More
Javascript/Greasemonkey: search for something then set result as a value

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok, I'm a bit of a n00b when it comes to JS (I'm not the greatest programmer) so please be gentle - specially if my questions been asked already somewhere and I'm too stupid to find the right answer. Self deprecation out of the way, let's get to the question. Problem There is a site me and a large… >>> More
Updating onclick's string value with Greasemonkey

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to write a Greasemonkey script to update the onclick value of a bunch of links on a page. The HTML looks like this: <a onclick="OpenGenericPopup('url-99.asp','popup',500,500,false)" href="javascript:void(0)">text</a> I need to update the url-99.asp part of the Javascript… >>> More

Related posts about crawler

Site crawler/spider that tosses results into mysql

as seen on Server Fault - Search for 'Server Fault'
It's been suggested that we use mysql for our site's search as it'd be running on the same server that hosts our web server (nginx) and our db (mysql). Since not all of our pages are created from the database, it's been suggested that we have a crawler that can crawl the site, and toss the page url… >>> More
Remove subdomain from Google Crawler

as seen on Server Fault - Search for 'Server Fault'
Hi all, I recently removed a sub-domain from my domain so I just have 1 website to manage. However, if I do a google search, my old domain is still there, I removed the sub-domain well over a week ago and if you try to access the domain directly, you will get an error saying the website can not… >>> More
Is there an automated way to take site inventory?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Is there a way to take site inventory using a crawler program that checks either the sources of images for specific servers that serve ads, or, that the crawler looks at a page for specific (html5?) tags like <aside> or some other tag to count the inventory of ad spaces available on a site?… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
What is a good Java crawler library?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am about to develop a crawler in Java but don't feel like reinventing the wheel. A quick Google search gives a whole bunch of Java libraries to build a web crawler. Besides that Nutch is of course a very robust package but seems a bit too advanced for my needs. I only need to crawl a handful… >>> More