Search Results

Search found 6 results on 1 pages for 'sberry2a'.

Page 1/1 | 1 

• Python, unit test - Pass command line arguments to setUp of unittest.TestCase

  - by sberry2A

  I have a script that acts as a wrapper for some unit tests written using the Python unittest module. In addition to cleaning up some files, creating an output stream and generating some code, it loads test cases into a suite using unittest.TestLoader().loadTestsFromTestCase() I am already using optparse to pull out several command-line arguments used for determining the output location, whether to regenerate code and whether to do some clean up. I also want to pass a configuration variable, namely an endpoint URI, for use within the test cases. I realize I can add an OptionParser to the setUp method of the TestCase, but I want to instead pass the option to setUp. Is this possible using loadTestsFromTestCase()? I can iterate over the returned TestSuite's TestCases, but can I manually call setUp on the TestCases? ** EDIT ** I wanted to point out that I am able to pass the arguments to setUp if I iterate over the tests and call setUp manually like: (options, args) = op.parse_args() suite = unittest.TestLoader().loadTestsFromTestCase(MyTests.TestSOAPFunctions) for test in suite: test.setUp(options.soap_uri) However, I am using xmlrunner for this and its run method takes a TestSuite as an argument. I assume it will run the setUp method itself, so I would need the parameters available within the XMLTestRunner. I hope this makes sense.

  Read the article

• Python: nonblocking read from stdout of threaded subprocess

  - by sberry2A

  I have a script (worker.py) that prints unbuffered output in the form... 1 2 3 . . . n where n is some constant number of iterations a loop in this script will make. In another script (service_controller.py) I start a number of threads, each of which starts a subprocess using subprocess.Popen(stdout=subprocess.PIPE, ...); Now, in my main thread (service_controller.py) I want to read the output of each thread's worker.py subprocess and use it to calculate an estimate for the time remaining till completion. I have all of the logic working that reads the stdout from worker.py and determines the last printed number. The problem is that I can not figure out how to do this in a non-blocking way. If I read a constant bufsize then each read will end up waiting for the same data from each of the workers. I have tried numerous ways including using fcntl, select + os.read, etc. What is my best option here? I can post my source if needed, but I figured the explanation describes the problem well enough. Thanks for any help here. EDIT Adding sample code I have a worker that starts a subprocess. class WorkerThread(threading.Thread): def __init__(self): self.completed = 0 self.process = None self.lock = threading.RLock() threading.Thread.__init__(self) def run(self): cmd = ["/path/to/script", "arg1", "arg2"] self.process = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1, shell=False) #flags = fcntl.fcntl(self.process.stdout, fcntl.F_GETFL) #fcntl.fcntl(self.process.stdout.fileno(), fcntl.F_SETFL, flags | os.O_NONBLOCK) def get_completed(self): self.lock.acquire(); fd = select.select([self.process.stdout.fileno()], [], [], 5)[0] if fd: self.data += os.read(fd, 1) try: self.completed = int(self.data.split("\n")[-2]) except IndexError: pass self.lock.release() return self.completed I then have a ThreadManager. class ThreadManager(): def __init__(self): self.pool = [] self.running = [] self.lock = threading.Lock() def clean_pool(self, pool): for worker in [x for x in pool is not x.isAlive()]: worker.join() pool.remove(worker) del worker return pool def run(self, concurrent=5): while len(self.running) + len(self.pool) > 0: self.clean_pool(self.running) n = min(max(concurrent - len(self.running), 0), len(self.pool)) if n > 0: for worker in self.pool[0:n]: worker.start() self.running.extend(self.pool[0:n]) del self.pool[0:n] time.sleep(.01) for worker in self.running + self.pool: worker.join() and some code to run it. threadManager = ThreadManager() for i in xrange(0, 5): threadManager.pool.append(WorkerThread()) threadManager.run() I have stripped out a log of the other code in hopes to try to pinpoint the issue.

  Read the article

• Python: Script works, but seems to deadlock after some time

  - by sberry2A

  I have the following script, which is working for the most part Link to PasteBin The script's job is to start a number of threads which in turn each start a subprocess with Popen. The output from each subprocess is as follows: 1 2 3 . . . n Done Bascially the subprocess is transferring 10M records from tables in one database to different tables in another db with a lot of data massaging/manipulation in between because of the different schemas. If the subprocess fails at any time in it's execution (bad records, duplicate primary keys, etc), or it completes successfully, it will output "Done\n". If there are no more records to select against for transfer then it will output "NO DATA\n" My intent was to create my script "tableTransfer.py" which would spawn a number of these processes, read their output, and in turn output information such as number of updates completed, time remaining, time elapsed, and number of transfers per second. I started running the process last night and checked in this morning to see it had deadlocked. There were not subprocceses running, there are still records to be updated, and the script had not exited. It was simply sitting there, no longer outputting the current information because no subprocces were running to update the total number complete which is what controls updates to the output. This is running on OS X. I am looking for three things: I would like to get rid of the possibility of this deadlock occurring so I don't need to check in on it as frequently. Is there some issue with locking? Am I doing this in a bad way (gThreading variable to control looping of spawning additional thread... etc.) I would appreciate some suggestions for improving my overall methodology. How should I handle ctrl-c exit? Right now I need to kill the process, but assume I should be able to use the signal module or other to catch the signal and kill the threads, is that right? I am not sure whether I should be pasting my entire script here, since I usually just paste snippets. Let me know if I should paste it here as well.

  Read the article

• Python: Created nested dictionary from list of paths

  - by sberry2A

  I have a list of tuples the looks similar to this (simplified here, there are over 14,000 of these tuples with more complicated paths than Obj.part) [ (Obj1.part1, {<SPEC>}), (Obj1.partN, {<SPEC>}), (ObjK.partN, {<SPEC>}) ] Where Obj goes from 1 - 1000, part from 0 - 2000. These "keys" all have a dictionary of specs associated with them which act as a lookup reference for inspecting another binary file. The specs dict contains information such as the bit offset, bit size, and C type of the data pointed to by the path ObjK.partN. For example: Obj4.part500 might have this spec, {'size':32, 'offset':128, 'type':'int'} which would let me know that to access Obj4.part500 in the binary file I must unpack 32 bits from offset 128. So, now I want to take my list of strings and create a nested dictionary which in the simplified case will look like this data = { 'Obj1' : {'part1':{spec}, 'partN':{spec} }, 'ObjK' : {'part1':{spec}, 'partN':{spec} } } To do this I am currently doing two things, 1. I am using a dotdict class to be able to use dot notation for dictionary get / set. That class looks like this: class dotdict(dict): def __getattr__(self, attr): return self.get(attr, None) __setattr__ = dict.__setitem__ __delattr__ = dict.__delitem__ The method for creating the nested "dotdict"s looks like this: def addPath(self, spec, parts, base): if len(parts) > 1: item = base.setdefault(parts[0], dotdict()) self.addPath(spec, parts[1:], item) else: item = base.setdefault(parts[0], spec) return base Then I just do something like: for path, spec in paths: self.lookup = dotdict() self.addPath(spec, path.split("."), self.lookup) So, in the end self.lookup.Obj4.part500 points to the spec. Is there a better (more pythonic) way to do this?

  Read the article

• Python: Unpack arbitary length bits for database storage

  - by sberry2A

  I have a binary data format consisting of 18,000+ packed int64s, ints, shorts, bytes and chars. The data is packed to minimize it's size, so they don't always use byte sized chunks. For example, a number whose min and max value are 31, 32 respectively might be stored with a single bit where the actual value is bitvalue + min, so 0 is 31 and 1 is 32. I am looking for the most efficient way to unpack all of these for subsequent processing and database storage. Right now I am able to read any value by using either struct.unpack, or BitBuffer. I use struct.unpack for any data that starts on a bit where (bit-offset % 8 == 0 and data-length % 8 == 0) and I use BitBuffer for anything else. I know the offset and size of every packed piece of data, so what is going to be the fasted way to completely unpack them? Many thanks.

  Read the article

• How to decode Google spreadsheet's Json respose as a Php Array

  - by Mohammad

  My google Docs Spreadsheet call returns this response in the json format (I only need everything after "rows") please look at the formatted response here : ) I use php's json_decode function to parse the data and use it (Yes, I am awful at php) This code returns NULL, and according to the documentation, NULL is returned "if the json cannot be decoded". $json = file_get_contents($jsonurl); $json_output = json_decode($json); var_dump ($json_output); // Returns NULL Basically, what i want to accomplish is to make a simple array from the first row values of the Json response. like this $array = {'john','John Handcock','[email protected]','2929292','blanc'} You guys are genius, I would appreciate your insight and help on this very much! Answer as "sberry2A" mentions bellow, the response is not valid Json, google offers the Zend Json library for this purpose, tho I decided to parse the tsv-excel version instead :)

  Read the article

1