How to reset Scrapy parameters? (always running under same parameters)

Posted by Jean Ventura on Stack Overflow See other posts from Stack Overflow or by Jean Ventura
Published on 2013-10-30T17:24:57Z Indexed on 2013/10/30 21:54 UTC
Read the original article Hit count: 256

Filed under:

python

|

twisted

|

scrapy

I've been running my Scrapy project with a couple of accounts (the project scrapes a especific site that requieres login credentials), but no matter the parameters I set, it always runs with the same ones (same credentials).

I'm running under virtualenv. Is there a variable or setting I'm missing?

Edit:

It seems that this problem is Twisted related.

Even when I run:

scrapy crawl -a user='user' -a password='pass' -o items.json -t json SpiderName

I still get an error saying:

ERROR: twisted.internet.error.ReactorNotRestartable

And all the information I get, is the last 'succesful' run of the spider.

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about twisted

Has Twisted changed its dependencies?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I'm currently working on a Python/Twisted project which is to be distributed and tested on Planetlab. For some reason my code was working on friday and now that I wanted to test a minor change it refuses to work at all: Traceback (most recent call last): File "acn_a4/src/node.py", line… >>> More
Python (Twisted) - reading from fifo and sending read data to multiple protocols

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, Im trying to write some kind of multi protocol bot (jabber/irc) that would read messages from fifo file (one liners mostly) and then send them to irc channel and jabber contacts. So far, I managed to create two factories to connect to jabber and irc, and they seem to be working. However, I've… >>> More
Start Python from Twisted

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I have learnt Python for about a month as a one year's PHPer.And I started from Twisted as I'm working in a corporation supplying webservice.I have finished some simple application such as data transferring service,page images-fetch service etc.But the problem is ,I don't understand the struture… >>> More
How to step through debug twisted?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'd like to be able to debug Punjab, a twisted python application, in Netbeans so that I can step through the code. How can I do that? Alternatively, how could I do it in a different debugger? >>> More
Python twisted Reactor class

as seen on Stack Overflow - Search for 'Stack Overflow'
What is the significance of the decorators @reactor.callWhenRunning, @results_deferred.addCallback @results_deferred.addErrback. Also what are deferred strings, for example in the twisted.internet.utils.getProcessOutput() returns a deferred string what exactly is happening here? I am new… >>> More