how to store data crawled from website
Posted
by Richard
on Stack Overflow
See other posts from Stack Overflow
or by Richard
Published on 2010-03-17T04:19:30Z
Indexed on
2010/03/17
4:31 UTC
Read the original article
Hit count: 214
I want to crawl a website and store the content on my computer for later analysis. However my OS file system has a limit on the number of sub directories, meaning storing the original folder structure is not going to work.
Suggestions?
Map the URL to some filename so can store flatly? Or just shove it in a database like sqlite to avoid file system limitations?
© Stack Overflow or respective owner