Python GUI Scraper hanging issues.
- by bball
I wrote a scraper using python a while back, and it worked fine in the command line. I have made a GUI for the application now, but I am having trouble with one issue. When I attempt to update text inside the gui (e.g. 'fetching URL 12/50'), I am unable seeing as the function within the scraper is grabbing 100+ links. Also when going from one scraping function, to a function that should update the gui, to another function, the gui update function seems to be skipped over while the next scrape function is run. An example would be:
scrapeLinksA() #takes 20 seconds
updateInfo("LinksA done")
scrapeLinksB() #takes another 20 seconds
in the above example, updateInfo is never executed, unless I end the program with a KeyboardInterrupt.
I'm thinking my solution is threading, but I'm not sure. What can I do to fix this?
I am using:
PyQt4
urllib2
BeautifulSoup