Message Queue: Which one is the best scenario?
- by pandaforme
I write a web crawler.
The crawler has 2 steps:
get a html page
then parse the page
I want to use message queue to improve performance and throughput.
I think 2 scenarios:
scenario 1:
structure:
urlProducer -> queue1 -> urlConsumer -> queue2 -> parserConsumer
urlProducer: get a target url and add it to queue1
…
