I'm interested in playing with this, anyone know of any code for this already existing? or what libraries would be useful to get started on something like this?
Hello. I've got the following code:
import time
import threading
class BaseWrapper: #static class
lock = threading.Lock()
@staticmethod
def synchronized_def():
BaseWrapper.lock.acquire()
time.sleep(5)
BaseWrapper.lock.release()
def test():
print time.ctime()
if __name__ is '__main__':
for i in xrange(10):
threading.Thread(target = test).start()
I want to have a method synchronized using static lock. However the above code prints the same time ten times, so it isn't really locking. How can I fix it? TIA
Hi i working on scrapy and trying xml feeds first time, below is my code
class TestxmlItemSpider(XMLFeedSpider):
name = "TestxmlItem"
allowed_domains = {"http://www.nasinteractive.com"}
start_urls = [
"http://www.nasinteractive.com/jobexport/advance/hcantexasexport.xml"
]
iterator = 'iternodes'
itertag = 'job'
def parse_node(self, response, node):
title = node.select('title/text()').extract()
job_code = node.select('job-code/text()').extract()
detail_url = node.select('detail-url/text()').extract()
category = node.select('job-category/text()').extract()
print title,";;;;;;;;;;;;;;;;;;;;;"
print job_code,";;;;;;;;;;;;;;;;;;;;;"
item = TestxmlItem()
item['title'] = node.select('title/text()').extract()
.......
return item
result:
File "/usr/lib/python2.7/site-packages/Scrapy-0.14.3-py2.7.egg/scrapy/item.py", line 56, in __setitem__
(self.__class__.__name__, key))
exceptions.KeyError: 'TestxmlItem does not support field: title'
Totally there are 200+ items so i need to loop over and assign the node text to item
but here all the results are displaying at once when we print, actually how can we loop over on nodes in scraping xml files with xmlfeedspider
Hi folks,
I'm using popen in order to send a few commands within a Django app.
Problem is that I'm getting [Error 5] Access Denied, apparently I have no access to cmd.exe, which popen seems to use.
WindowsError at /test/cmd/
[Error 5] Access is denied: 'C:\WINDOWS\system32\cmd.exe /c dir'
I reckon this is because the app sits behind a web server which has limited privileges.
Is there anything we can do about it? Help would be awesome!
I'm trying to import a module, while passing some global variables, but it does not seem to work:
File test_1:
test_2 = __import__("test_2", {"testvar": 1})
File test_2:
print testvar
This seems like it should work, and print a 1, but I get the following error when I run test_1:
Traceback (most recent call last):
File ".../test_1.py", line 1, in <module>
print testvar
NameError: name 'testvar' is not defined
What am I doing wrong?
Ok, so the title may trick you a bit, and I'm sorry for that but didn't find a better title.
This question might be a bit hard to understand so I'll try my best.
I have no idea how this works or if it is even possible but what I want to do is for example create a file type (lets imagine .test (in which a random file name would be random.test)). Now before I continue, its obviously easy to do this using for example:
filename = "random.test"
file = open(filename, 'w')
file.write("some text here")
But now what I would like to know is if it is possible to write the file .test so if I set it to open with a wxPython program, it recognizes it and for example opens up a Message Dialog automatically.
I'm sorry if I'm being vague and in case you don't understand, let me know so I can try my best to explain you.
How can I do this with Firefox?
ie = win32com.client.Dispatch('InternetExplorer.Application')
ie.visible = 1
ie.navigate('http://google.com')
Is there a way to do it?
Hello guys
I have this code, and I would like to use the app parameter to generate the code instead of duplicating it.
if app == 'map':
try:
from modulo.map.views import map
return map(request, *args, **kwargs)
except ImportError:
pass
elif app == 'schedule':
try:
from modulo.schedule.views import schedule_day
return schedule_day(request, *args, **kwargs)
except ImportError:
pass
elif app == 'sponsors':
try:
from modulo.sponsors.views import sponsors
return sponsors(request, *args, **kwargs)
except ImportError:
pass
elif app == 'streaming':
try:
from modulo.streaming.views import streaming
return streaming(request, *args, **kwargs)
except ImportError:
pass
Do you have any idea ?
Thanks
When I try to write a field that includes whitespace in it, it gets split into multiple fields on the space. What's causing this? It's driving me insane. Thanks
data = open("file.csv", "wb")
w = csv.writer(data)
w.writerow(['word1', 'word2'])
w.writerow(['word 1', 'word2'])
data.close()
I'll get 2 fields(word1,word2) for first example and 3(word,1,word2) for the second.
Today I was bitten again by "Mutable default arguments" after many years. I usually don't use mutable default arguments unless needed but I think with time I forgot about that, and today in the application I added tocElements=[] in a pdf generation function's argument list and now 'Table of Content' gets longer and longer after each invocation of "generate pdf" :)
My question is what other things should I add to my list of things to MUST avoid?
1 Mutable default arguments
2 import modules always same way e.g. 'from y import x' and 'import x' are totally different things actually they are treated as different modules
see http://stackoverflow.com/questions/1459236/module-reimported-if-imported-from-different-path
3 Do not use range in place of lists because range() will become an iterator anyway, so things like this will fail, so wrap it by list
myIndexList = [0,1,3]
isListSorted = myIndexList == range(3) # will fail in 3.0
isListSorted = myIndexList == list(range(3)) # will not
same thing can be mistakenly done with xrange e.g myIndexList == xrange(3).
4 Catching multiple exceptions
try:
raise KeyError("hmm bug")
except KeyError,TypeError:
print TypeError
It prints "hmm bug", though it is not a bug, it looks like we are catching exceptions of type KeyError,TypeError but instead we are catching KeyError only as variable TypeError, instead use
try:
raise KeyError("hmm bug")
except (KeyError,TypeError):
print TypeError
What am I doing wrong to get this error?
replacements = {}
replacements["**"] = ("<strong>", "</strong>")
replacements["__"] = ("<em>", "</em>")
replacements["--"] = ("<blink>", "</blink>")
replacements["=="] = ("<marquee>", "</marquee>")
replacements["@@"] = ("<code>", "</code>")
for delimiter, (open_tag, close_tag) in replacements: # error here
message = self.replaceFormatting(delimiter, message, open_tag, close_tag);
The error:
Traceback (most recent call last):
File "", line 1, in
for shit, (a, b) in replacements: ValueError: need more than 1 value to
unpack
All the values tuples have two values. Right?
I need to open multiple files (2 input and 2 output files), do complex manipulations on the lines from input files and then append results at the end of 2 output files. I am currently using the following approach:
in_1 = open(input_1)
in_2 = open(input_2)
out_1 = open(output_1, "w")
out_2 = open(output_2, "w")
# Read one line from each 'in_' file
# Do many operations on the DNA sequences included in the input files
# Append one line to each 'out_' file
in_1.close()
in_2.close()
out_1.close()
out_2.close()
The files are huge (each potentially approaching 1Go, that is why I am reading through these input files one at a time. I am guessing that this is not a very Pythonic way to do things. :) Would using the following form good?
with open("file1") as f1:
with open("file2") as f2: # etc.
If yes, could I do it while avoiding the highly indented code that would result? Thanks for the insights!
I have a class which is being operated on by two functions. One function creates a list of widgets and writes it into the class:
def updateWidgets(self):
widgets = self.generateWidgetList()
self.widgets = widgets
the other function deals with the widgets in some way:
def workOnWidgets(self):
for widget in self.widgets:
self.workOnWidget(widget)
each of these functions runs in it's own thread. the question is, what happens if the updateWidgets() thread executes while the workOnWidgets() thread is running?
I am assuming that the iterator created as part of the for...in loop will keep some kind of reference to the old self.widgets object? So I will finish iterating over the old list... but I'd love to know for sure.
Hi,
I want to implement a lightweight Message Queue proxy. It's job is to receive messages from a web application (PHP) and send them to the Message Queue server asynchronously. The reason for this proxy is that the MQ isn't always avaliable and is sometimes lagging, or even down, but I want to make sure the messages are delivered, and the web application returns immediately.
So, PHP would send the message to the MQ proxy running on the same host. That proxy would save the messages to SQLite for persistence, in case of crashes. At the same time it would send the messages from SQLite to the MQ in batches when the connection is available, and delete them from SQLite.
Now, the way I understand, there are these components in this service:
message listener (listens to the messages from PHP and writes them to a Incoming Queue)
DB flusher (reads messages from the Incoming Queue and saves them to a database; due to SQLite single-threadedness)
MQ connection handler (keeps the connection to the MQ server online by reconnecting)
message sender (collects messages from SQlite db and sends them to the MQ server, then removes them from db)
I was thinking of using Twisted for #1 (TCPServer), but I'm having problem with integrating it with other points, which aren't event-driven. Intuition tells me that each of these points should be running in a separate thread, because all are IO-bound and independent of each other, but I could easily put them in a single thread. Even though, I couldn't find any good and clear (to me) examples on how to implement this worker thread aside of Twisted's main loop.
The example I've started with is the chatserver.py, which uses service.Application and internet.TCPServer objects. If I start my own thread prior to creating TCPServer service, it runs a few times, but the it stops and never runs again. I'm not sure, why this is happening, but it's probably because I don't use threads with Twisted correctly.
Any suggestions on how to implement a separate worker thread and keep Twisted? Do you have any alternative architectures in mind?
I have a directory /tmp/dir with two types of file names
/tmp/dir/abc-something-server.log
/tmp/dir/xyz-something-server.log
..
..
and
/tmp/dir/something-client.log
I need append a few lines (these lines are constant) to files end with "client.log"
line 1
line 2
line 3
line 4
append these four lines to files end with "client.log"
and For files end with "server.log"
I needed to append after a keyword say "After-this".
"server.log " file has multiple entries of "After-this" I need to find the first entry of "After-this" and append above said four lines keep the remaining file as it is.
Any help will be great appreciated :) Thanks in advance.
I have an HTML table that I'd like to be able to export to an Excel file. I already have an option to export the table into an IQY file, but I'd prefer something that didn't allow the user to refresh the data via Excel. I just want a feature that takes a snapshot of the table at the time the user clicks the link/button.
I'd prefer it if the feature was a link/button on the HTML page that allows the user to save the query results displayed in the table. Is there a way to do this at all? Or, something I can modify with the IQY?
I can try to provide more details if needed. Thanks in advance.
I am trying to install a package via pip, but there were missing files from the zip file. So I copy the files and then compile with gcc. But now I cannot continue with the installation by calling pip install because it sees a pre-existing directory and will not proceed.
This is with pip version 1.5.6, but I thought that with earlier versions of pip that it was less fussy about this.
What are the remaining steps to complete the package installation?
Consider this situation:
I get an object of type A which has the function f. I.e:
class A:
def f():
print 'in f'
def h():
print 'in h'
and I get an instance of this class but I want to override the f function but save the rest of the functionality of A. So what I was thinking was something of the sort:
class B(A):
....
def f():
print 'in B->f'
and the usage would be:
def main(a):
b = B(a)
b.f() #prints "in B->f"
b.h() #print "in h"
How do you do such a thing?
I have a hashmap like so:
results[tweet_id] = {"score" : float(dot(query,doc) / (norm(query) * norm(doc))), "tweet" : tweet}
What I'd like to do is to sort results by the innser "score" key. I don't know how possible this is, I saw many sorting tutorials but they were for simple (not nested) data structures.
I'm not even sure what the right words are to search for. I want to display parts of the error object in an except block (similar to the err object in VBScript, which has Err.Number and Err.Description). For example, I want to show the values of my variables, then show the exact error. Clearly, I am causing a divided-by-zero error below, but how can I print that fact?
try:
x = 0
y = 1
z = y / x
z = z + 1
print "z=%d" % (z)
except:
print "Values at Exception: x=%d y=%d " % (x,y)
print "The error was on line ..."
print "The reason for the error was ..."
Can someone explain help me understand how the this bit of code works? Particularly how the myHeap assignment works. I know the freq variable is assigned as a dictionary. But what about my myHeap? is it a Set?
exe_Data = {
'e' : 0.124167,
't' : 0.0969225,
'a' : 0.0820011,
'i' : 0.0768052,
}
freq = exe_Data)
myHeap = [[pct, [symbol, ""]] for symbol, pct in freq.items()]
Hi folks,
is there a similar library to simplejson, which would enable quick serialization of data to and from XML.
e.g. json.loads('{vol:'III', title:'Magical Unicorn'}')
e.g. json.dumps([1,2,3,4,5])
Any ideas?
Hi
I have this code that fetches some text from a page using BeautifulSoup
soup= BeautifulSoup(html)
body = soup.find('div' , {'id':'body'})
print body
I would like to make this as a reusable function that takes in some htmltext and the tags to match it like the following
def parse(html, atrs):
soup= BeautifulSoup(html)
body = soup.find(atrs)
return body
But if i make a call like this
parse(htmlpage, ('div' , {'id':'body'}")) or like
parse(htmlpage, ['div' , {'id':'body'}"])
I get only the div element, the body attribute seems to get ignored.
Is there a way to fix this?
say ive got a matrix that looks like:
[[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]
how can i make it on seperate lines:
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]
and then remove commas etc:
0 0 0 0 0
And also to make it blank instead of 0's, so that numbers can be put in later, so in the end it will be like:
_ 1 2 _ 1 _ 1
(spaces not underscores)
thanks