Search Results

Search found 92 results on 4 pages for 'multiprocessing'.

Page 1/4 | 1 2 3 4  | Next Page >

• Python Multiprocessing with Queue vs ZeroMQ IPC

  - by Imraan

  I am busy writing a Python application using ZeroMQ and implementing a variation of the Majordomo pattern as described in the ZGuide. I have a broker as an intermediary between a set of workers and clients. I want to do some extensive logging for every request that comes in, but I do not want the broker to waste time doing that. The broker should pass that logging request to something else. I have thought of two ways :- Create workers that are only for logging and use the ZeroMQ IPC transport Use Multiprocessing with a Queue I am not sure which one is better or faster for that matter. The first option does allow me to use the current worker base classes that I already use for normal workers, but the second option seems quicker to implement. I would like some advice or comments on the above or possibly a different solution.

  Read the article

• Python multiprocessing global variable updates not returned to parent

  - by user1459256

  I am trying to return values from subprocesses but these values are unfortunately unpicklable. So I used global variables in threads module with success but have not been able to retrieve updates done in subprocesses when using multiprocessing module. I hope I'm missing something. The results printed at the end are always the same as initial values given the vars dataDV03 and dataDV04. The subprocesses are updating these global variables but these global variables remain unchanged in the parent. import multiprocessing # NOT ABLE to get python to return values in passed variables. ants = ['DV03', 'DV04'] dataDV03 = ['', ''] dataDV04 = {'driver': '', 'status': ''} def getDV03CclDrivers(lib): # call global variable global dataDV03 dataDV03[1] = 1 dataDV03[0] = 0 # eval( 'CCL.' + lib + '.' + lib + '( "DV03" )' ) these are unpicklable instantiations def getDV04CclDrivers(lib, dataDV04): # pass global variable dataDV04['driver'] = 0 # eval( 'CCL.' + lib + '.' + lib + '( "DV04" )' ) if __name__ == "__main__": jobs = [] if 'DV03' in ants: j = multiprocessing.Process(target=getDV03CclDrivers, args=('LORR',)) jobs.append(j) if 'DV04' in ants: j = multiprocessing.Process(target=getDV04CclDrivers, args=('LORR', dataDV04)) jobs.append(j) for j in jobs: j.start() for j in jobs: j.join() print 'Results:\n' print 'DV03', dataDV03 print 'DV04', dataDV04 I cannot post to my question so will try to edit the original. Here is the object that is not picklable: In [1]: from CCL import LORR In [2]: lorr=LORR.LORR('DV20', None) In [3]: lorr Out[3]: <CCL.LORR.LORR instance at 0x94b188c> This is the error returned when I use a multiprocessing.Pool to return the instance back to the parent: Thread getCcl (('DV20', 'LORR'),) Process PoolWorker-1: Traceback (most recent call last): File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap self.run() File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/pool.py", line 71, in worker put((job, i, result)) File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/queues.py", line 366, in put return send(obj) UnpickleableError: Cannot pickle <type 'thread.lock'> objects In [5]: dir(lorr) Out[5]: ['GET_AMBIENT_TEMPERATURE', 'GET_CAN_ERROR', 'GET_CAN_ERROR_COUNT', 'GET_CHANNEL_NUMBER', 'GET_COUNT_PER_C_OP', 'GET_COUNT_REMAINING_OP', 'GET_DCM_LOCKED', 'GET_EFC_125_MHZ', 'GET_EFC_COMB_LINE_PLL', 'GET_ERROR_CODE_LAST_CAN_ERROR', 'GET_INTERNAL_SLAVE_ERROR_CODE', 'GET_MAGNITUDE_CELSIUS_OP', 'GET_MAJOR_REV_LEVEL', 'GET_MINOR_REV_LEVEL', 'GET_MODULE_CODES_CDAY', 'GET_MODULE_CODES_CMONTH', 'GET_MODULE_CODES_DIG1', 'GET_MODULE_CODES_DIG2', 'GET_MODULE_CODES_DIG4', 'GET_MODULE_CODES_DIG6', 'GET_MODULE_CODES_SERIAL', 'GET_MODULE_CODES_VERSION_MAJOR', 'GET_MODULE_CODES_VERSION_MINOR', 'GET_MODULE_CODES_YEAR', 'GET_NODE_ADDRESS', 'GET_OPTICAL_POWER_OFF', 'GET_OUTPUT_125MHZ_LOCKED', 'GET_OUTPUT_2GHZ_LOCKED', 'GET_PATCH_LEVEL', 'GET_POWER_SUPPLY_12V_NOT_OK', 'GET_POWER_SUPPLY_15V_NOT_OK', 'GET_PROTOCOL_MAJOR_REV_LEVEL', 'GET_PROTOCOL_MINOR_REV_LEVEL', 'GET_PROTOCOL_PATCH_LEVEL', 'GET_PROTOCOL_REV_LEVEL', 'GET_PWR_125_MHZ', 'GET_PWR_25_MHZ', 'GET_PWR_2_GHZ', 'GET_READ_MODULE_CODES', 'GET_RX_OPT_PWR', 'GET_SERIAL_NUMBER', 'GET_SIGN_OP', 'GET_STATUS', 'GET_SW_REV_LEVEL', 'GET_TE_LENGTH', 'GET_TE_LONG_FLAG_SET', 'GET_TE_OFFSET_COUNTER', 'GET_TE_SHORT_FLAG_SET', 'GET_TRANS_NUM', 'GET_VDC_12', 'GET_VDC_15', 'GET_VDC_7', 'GET_VDC_MINUS_7', 'SET_CLEAR_FLAGS', 'SET_FPGA_LOGIC_RESET', 'SET_RESET_AMBSI', 'SET_RESET_DEVICE', 'SET_RESYNC_TE', 'STATUS', '_HardwareDevice__componentName', '_HardwareDevice__hw', '_HardwareDevice__stickyFlag', '_LORRBase__logger', '__del__', '__doc__', '__init__', '__module__', '_devices', 'clearDeviceCommunicationErrorAlarm', 'getControlList', 'getDeviceCommunicationErrorCounter', 'getErrorMessage', 'getHwState', 'getInternalSlaveCanErrorMsg', 'getLastCanErrorMsg', 'getMonitorList', 'hwConfigure', 'hwDiagnostic', 'hwInitialize', 'hwOperational', 'hwSimulation', 'hwStart', 'hwStop', 'inErrorState', 'isMonitoring', 'isSimulated'] In [6]:

  Read the article

• Multiprocessing Bomb

  - by iKarampa

  I was working the following example from Doug Hellmann tutorial on multiprocessing: import multiprocessing def worker(): """worker function""" print 'Worker' return if __name__ == '__main__': jobs = [] for i in range(5): p = multiprocessing.Process(target=worker) jobs.append(p) p.start() When I tried to run it outside the if statement: import multiprocessing def worker(): """worker function""" print 'Worker' jobs = [] for i in range(5): p = multiprocessing.Process(target=worker) jobs.append(p) p.start() It started spawning processes non-stop, without any way of to terminating it. Why would that happen? Why it did not generate 5 processes and exit? Why do I need the if statement?

  Read the article

• yet another confusion with multiprocessing error, 'module' object has no attribute 'f'

  - by gatoatigrado

  I know this has been answered before, but it seems that executing the script directly "python filename.py" does not work. I have Python 2.6.2 on SuSE Linux. Code: #!/usr/bin/python # -*- coding: utf-8 -*- from multiprocessing import Pool p = Pool(1) def f(x): return x*x p.map(f, [1, 2, 3]) Command line: > python example.py Process PoolWorker-1: Traceback (most recent call last): File "/usr/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap self.run() File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker task = get() File "/usr/lib/python2.6/multiprocessing/queues.py", line 339, in get return recv() AttributeError: 'module' object has no attribute 'f'

  Read the article

• Python - Help with multiprocessing / threading basics.

  - by orokusaki

  I haven't ever used multi-threading, and I decided to learn it today. I was reluctant to ever use it before, but when I tried it out it seemed way to easy, which makes me wary. Are there any gotchas in my code, or is it really that simple? import uuid import time import multiprocessing def sleep_then_write(content): time.sleep(5) f = open(unicode(uuid.uuid4()), 'w') f.write(content) f.close() if __name__ == '__main__': for i in range(3): p = multiprocessing.Process(target=sleep_then_write, args=('Hello World',)) p.start() My primary purpose of using threading would be to offload multiple images to S3 after re-sizing them, all at the same time. Is that a reasonable task for Python's multiprocessing? I've read a lot about certain types of tasks not really getting any gain from using threading in Python due to the GIL, but it seems that multiprocessing completely removes that worry, yes? I can imagine a case where 50 users hit the system and it spawns 150 Python interpreters. I can also imagine that wouldn't be good on a production server. How can something like that be avoided? Finally (but most important): How can I return control back to the caller of the new processes? I need to be able to continue with returning an HTTP response and content back to the user and then have the processes continue doing there work after the user of my website is done with the transaction.

  Read the article

• Python multiprocessing doesn't play nicely with uuid.uuid4().

  - by yig

  I'm trying to generate a uuid for a filename, and I'm also using the multiprocessing module. Unpleasantly, all of my uuids end up exactly the same. Here is a small example: import multiprocessing import uuid def get_uuid( a ): ## Doesn't help to cycle through a bunch. #for i in xrange(10): uuid.uuid4() ## Doesn't help to reload the module. #reload( uuid ) ## Doesn't help to load it at the last minute. ## (I simultaneously comment out the module-level import). #import uuid ## uuid1() does work, but it differs only in the first 8 characters and includes identifying information about the computer. #return uuid.uuid1() return uuid.uuid4() def main(): pool = multiprocessing.Pool( 20 ) uuids = pool.map( get_uuid, range( 20 ) ) for id in uuids: print id if __name__ == '__main__': main() I peeked into uuid.py's code, and it seems to depending-on-the-platform use some OS-level routines for randomness, so I'm stumped as to a python-level solution (to do something like reload the uuid module or choose a new random seed). I could use uuid.uuid1(), but only 8 digits differ and I think there are derived exclusively from the time, which seems dangerous especially given that I'm multiprocessing (so the code could be executing at exactly the same time). Is there some Wisdom out there about this issue?

  Read the article

• How to synchronize a python dict with multiprocessing

  - by Peter Smit

  I am using Python 2.6 and the multiprocessing module for multi-threading. Now I would like to have a synchronized dict (where the only atomic operation I really need is the += operator on a value). Should I wrap the dict with a multiprocessing.sharedctypes.synchronized() call? Or is another way the way to go?

  Read the article

• Memory leak appears only when multiprocessing

  - by Sandro

  I am trying to use python's multiprocessing library to hopefully gain some performance. Specifically I am using its map function. Now, for some reason when I swap it out with its single threaded counterpart I don't get any memory leaks over time. But using the multiprocessing version of map causes my memory to go through the roof. For the record I am doing something which can easily hog up loads of memory, but what would the difference be between the two to cause such a stark difference?

  Read the article

• Multiprocessing vs Threading Python

  - by John

  Hello, I am trying to understand the advantages of the module Multiprocessing over Threading. I know that Multiprocessing get's around the Global Interpreter Lock, but what other advantages are there, and can threading not do the same thing?

  Read the article

• Solving embarassingly parallel problems using Python multiprocessing

  - by gotgenes

  How does one use multiprocessing to tackle embarrassingly parallel problems? Embarassingly parallel problems typically consist of three basic parts: Read input data (from a file, database, tcp connection, etc.). Run calculations on the input data, where each calculation is independent of any other calculation. Write results of calculations (to a file, database, tcp connection, etc.). We can parallelize the program in two dimensions: Part 2 can run on multiple cores, since each calculation is independent; order of processing doesn't matter. Each part can run independently. Part 1 can place data on an input queue, part 2 can pull data off the input queue and put results onto an output queue, and part 3 can pull results off the output queue and write them out. This seems a most basic pattern in concurrent programming, but I am still lost in trying to solve it, so let's write a canonical example to illustrate how this is done using multiprocessing. Here is the example problem: Given a CSV file with rows of integers as input, compute their sums. Separate the problem into three parts, which can all run in parallel: Process the input file into raw data (lists/iterables of integers) Calculate the sums of the data, in parallel Output the sums Below is traditional, single-process bound Python program which solves these three tasks: #!/usr/bin/env python # -*- coding: UTF-8 -*- # basicsums.py """A program that reads integer values from a CSV file and writes out their sums to another CSV file. """ import csv import optparse import sys def make_cli_parser(): """Make the command line interface parser.""" usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV", __doc__, """ ARGUMENTS: INPUT_CSV: an input CSV file with rows of numbers OUTPUT_CSV: an output file that will contain the sums\ """]) cli_parser = optparse.OptionParser(usage) return cli_parser def parse_input_csv(csvfile): """Parses the input CSV and yields tuples with the index of the row as the first element, and the integers of the row as the second element. The index is zero-index based. :Parameters: - `csvfile`: a `csv.reader` instance """ for i, row in enumerate(csvfile): row = [int(entry) for entry in row] yield i, row def sum_rows(rows): """Yields a tuple with the index of each input list of integers as the first element, and the sum of the list of integers as the second element. The index is zero-index based. :Parameters: - `rows`: an iterable of tuples, with the index of the original row as the first element, and a list of integers as the second element """ for i, row in rows: yield i, sum(row) def write_results(csvfile, results): """Writes a series of results to an outfile, where the first column is the index of the original row of data, and the second column is the result of the calculation. The index is zero-index based. :Parameters: - `csvfile`: a `csv.writer` instance to which to write results - `results`: an iterable of tuples, with the index (zero-based) of the original row as the first element, and the calculated result from that row as the second element """ for result_row in results: csvfile.writerow(result_row) def main(argv): cli_parser = make_cli_parser() opts, args = cli_parser.parse_args(argv) if len(args) != 2: cli_parser.error("Please provide an input file and output file.") infile = open(args[0]) in_csvfile = csv.reader(infile) outfile = open(args[1], 'w') out_csvfile = csv.writer(outfile) # gets an iterable of rows that's not yet evaluated input_rows = parse_input_csv(in_csvfile) # sends the rows iterable to sum_rows() for results iterable, but # still not evaluated result_rows = sum_rows(input_rows) # finally evaluation takes place as a chain in write_results() write_results(out_csvfile, result_rows) infile.close() outfile.close() if __name__ == '__main__': main(sys.argv[1:]) Let's take this program and rewrite it to use multiprocessing to parallelize the three parts outlined above. Below is a skeleton of this new, parallelized program, that needs to be fleshed out to address the parts in the comments: #!/usr/bin/env python # -*- coding: UTF-8 -*- # multiproc_sums.py """A program that reads integer values from a CSV file and writes out their sums to another CSV file, using multiple processes if desired. """ import csv import multiprocessing import optparse import sys NUM_PROCS = multiprocessing.cpu_count() def make_cli_parser(): """Make the command line interface parser.""" usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV", __doc__, """ ARGUMENTS: INPUT_CSV: an input CSV file with rows of numbers OUTPUT_CSV: an output file that will contain the sums\ """]) cli_parser = optparse.OptionParser(usage) cli_parser.add_option('-n', '--numprocs', type='int', default=NUM_PROCS, help="Number of processes to launch [DEFAULT: %default]") return cli_parser def main(argv): cli_parser = make_cli_parser() opts, args = cli_parser.parse_args(argv) if len(args) != 2: cli_parser.error("Please provide an input file and output file.") infile = open(args[0]) in_csvfile = csv.reader(infile) outfile = open(args[1], 'w') out_csvfile = csv.writer(outfile) # Parse the input file and add the parsed data to a queue for # processing, possibly chunking to decrease communication between # processes. # Process the parsed data as soon as any (chunks) appear on the # queue, using as many processes as allotted by the user # (opts.numprocs); place results on a queue for output. # # Terminate processes when the parser stops putting data in the # input queue. # Write the results to disk as soon as they appear on the output # queue. # Ensure all child processes have terminated. # Clean up files. infile.close() outfile.close() if __name__ == '__main__': main(sys.argv[1:]) These pieces of code, as well as another piece of code that can generate example CSV files for testing purposes, can be found on github. I would appreciate any insight here as to how you concurrency gurus would approach this problem. Here are some questions I had when thinking about this problem. Bonus points for addressing any/all: Should I have child processes for reading in the data and placing it into the queue, or can the main process do this without blocking until all input is read? Likewise, should I have a child process for writing the results out from the processed queue, or can the main process do this without having to wait for all the results? Should I use a processes pool for the sum operations? If yes, what method do I call on the pool to get it to start processing the results coming into the input queue, without blocking the input and output processes, too? apply_async()? map_async()? imap()? imap_unordered()? Suppose we didn't need to siphon off the input and output queues as data entered them, but could wait until all input was parsed and all results were calculated (e.g., because we know all the input and output will fit in system memory). Should we change the algorithm in any way (e.g., not run any processes concurrently with I/O)?

  Read the article

• Comparison of the multiprocessing module and pyro?

  - by fivebells

  I use pyro for basic management of parallel jobs on a compute cluster. I just moved to a cluster where I will be responsible for using all the cores on each compute node. (On previous clusters, each core has been a separate node.) The python multiprocessing module seems like a good fit for this. I notice it can also be used for remote-process communication. If anyone has used both frameworks for remote-process communication, I'd be grateful to hear how they stack up against each other. The obvious benefit of the multiprocessing module is that it's built-in from 2.6. Apart from that, it's hard for me to tell which is better.

  Read the article

• Python - multithreading / multiprocessing, very strange problem.

  - by orokusaki

  import uuid import time import multiprocessing def sleep_then_write(content): time.sleep(5) print(content) if __name__ == '__main__': for i in range(15): p = multiprocessing.Process(target=sleep_then_write, args=('Hello World',)) p.start() print('Ah, what a hard day of threading...') This script output the following: Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... AAh, what a hard day of threading.. h, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Ah, what a hard day of threading... Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Hello World Firstly, why the heck did it print the bottom statement sixteen times (one for each process) instead of just the one time? Second, notice the AAh, and h, about half way down; that was the real output. This makes me wary of using threads ever, now. (Windows XP, Python 2.6.4, Core 2 Duo)

  Read the article

• Generate and merge data with python multiprocessing

  - by Bobby

  I have a list of starting data. I want to apply a function to the starting data that creates a few pieces of new data for each element in the starting data. Some pieces of the new data are the same and I want to remove them. The sequential version is essentially: def create_new_data_for(datum): """make a list of new data from some old datum""" return [datum.modified_copy(k) for k in datum.k_list] data = [some list of data] #some data to start with #generate a list of new data from the old data, we'll reduce it next newdata = [] for d in data: newdata.extend(create_new_data_for(d)) #now reduce the data under ".matches(other)" reduced = [] for d in newdata: for seen in reduced: if d.matches(seen): break #so we haven't seen anything like d yet seen.append(d) #now reduced is finished and is what we want! I want to speed this up with multiprocessing. I was thinking that I could use a multiprocessing.Queue for the generation. Each process would just put the stuff it creates on, and when the processes are reducing the data, they can just get the data from the Queue. But I'm not sure how to have the different process loop over reduced and modify it without any race conditions or other issues. What is the best way to do this safely? or is there a different way to accomplish this goal better?

  Read the article

• How to pick a chunksize for python multiprocessing with large datasets

  - by Sandro

  I am attempting to to use python to gain some performance on a task that can be highly parallelized using http://docs.python.org/library/multiprocessing. When looking at their library they say to use chunk size for very long iterables. Now, my iterable is not long, one of the dicts that it contains is huge: ~100000 entries, with tuples as keys and numpy arrays for values. How would I set the chunksize to handle this and how can I transfer this data quickly? Thank you.

  Read the article

• python multiprocessing member variable not set

  - by Jake

  In the following script, I get the "stop message received" output but the process never ends. Why is that? Is there another way to end a process besides terminate or os.kill that is along these lines? from multiprocessing import Process from time import sleep class Test(Process): def __init__(self): Process.__init__(self) self.stop = False def run(self): while self.stop == False: print "running" sleep(1.0) def end(self): print "stop message received" self.stop = True if __name__ == "__main__": test = Test() test.start() sleep(1.0) test.end() test.join()

  Read the article

• Sending and receiving async over multiprocessing.Pipe() in Python

  - by dcolish

  I'm having some issues getting the Pipe.send to work in this code. What I would ultimately like to do is send and receive messages to and from the foreign process while its running in a fork. This is eventually going to be integrated into a pexpect loop for talking to interpreter processes. ` from multiprocessing import Process, Pipe def f(conn): cmd = '' if conn.poll(): cmd = conn.recv() i = 1 i += 1 conn.send([42 + i, cmd, 'hello']) if __name__ == '__main__': parent_conn, child_conn = Pipe() p = Process(target=f, args=(child_conn,)) p.start() from pdb import set_trace; set_trace() while parent_conn.poll(): print parent_conn.recv() # prints "[42, None, 'hello']" parent_conn.send('OHHAI') p.join() `

  Read the article

• python multiprocessing.Process.Manager not producing consistent results?

  - by COpython

  I've written the following code to illustrate the problem I'm seeing. I'm trying to use a Process.Manager.list() to keep track of a list and increment random indices of that list. Each time there are 100 processes spawned, and each process increments a random index of the list by 1. Therefore, one would expect the SUM of the resulting list to be the same each time, correct? I get something between 203 and 205. from multiprocessing import Process, Manager import random class MyProc(Process): def __init__(self, A): Process.__init__(self) self.A = A def run(self): i = random.randint(0, len(self.A)-1) self.A[i] = self.A[i] + 1 if __name__ == '__main__': procs = [] M = Manager() a = M.list(range(15)) print('A: {0}'.format(a)) print('sum(A) = {0}'.format(sum(a))) for i in range(100): procs.append(MyProc(a)) map(lambda x: x.start(), procs) map(lambda x: x.join(), procs) print('A: {0}'.format(a)) print('sum(A) = {0}'.format(sum(a)))

  Read the article

• wxpython GUI and multiprocessing - how to send data back from the long running process

  - by wxpydon

  Hello everyone, Trying to run a time consuming task from a wxpython GUI. The basic idea is to start the long time task from the GUI (pressing a button) and then, a static text on the dialog should be updated from it. First I tried some threading (http://wiki.wxpython.org/LongRunningTasks and many other resourses seen), and I want to show back the messages using Publisher.class. It didn't went so well, after a message or two, the GUI seems to frozen. Now I want to achieve that with multiprocessing. I have this method inside my 'GUI' class: def do_update(self, e): self.txt_updatemsg.SetLabel("Don't stop this \n") ... pub = Publisher() # i tried also calling directly from dbob object # Publisher() is a singleton so this must be useless? pub.subscribe(self.__update_txt_message, ('updatedlg', 'message')) dbob = dbutils.DBUtils() # DBUtils is the class with 'long time' tasks dbob.publisher = pub p = Process(target=self.do_update_process, args=(dbob,)) p.start() while p.is_alive: wx.Yield def do_update_process(self, dbob): dbob.do_update() __update_txt_message is a simple function what sets the static text on dialog. Question is: how can I send back some text messages from this Process (just simple text, that's all I need) Thanks guys!

  Read the article

• Python Multiprocessing Question

  - by Avan

  My program has 2 parts divided into the core and downloader. The core handles all the app logic while the downloader just downloads urls. Right now, I am trying to use the python multiprocessing module to accomplish the task of the core as a process and the downloader as a process. The first problem I noticed was that if I spawn the downloader process from the core process so that the downloader is the child process and the core is the parent, the core process(parent) is blocked until the child is finished. I do not want this behaivor though. I would like to have a core process and a downloader process that are both able to execute their code and communicate between each other. example ... def main(): jobQueue = Queue() jobQueue.put("http://google.com) d = Downloader(jobQueue) p = Process(target=d.start()) p.start() if __name__ == '__main__': freeze_support() main() where Downloader's start() just takes the url out of the queue and downloads it. In order to have the 2 processes unblocked, would I need to create 2 processes from the parent process and then share something between them?

  Read the article

• Django ORM and multiprocessing

  - by Ankur Gupta

  Hi, I am using Django ORM in my python script in a decoupled fashion i.e. it's not running in context of a normal Django Project. I am also using the multi processing module. And different process in turn are making queries. The process ran successfully for an hr and exited with this message "IOError: [Errno 32] Broken pipe" Upon futhur diagnosis and debugging this error pops up when I call save() on the model instance. I am wondering Is Django ORM Process save ? Why would this error arise else ? Cheers Ankur

  Read the article

• python multiprocessing server connections

  - by tranimatronic

  I wish to get a list of connections to a manager. I can get last_accepted from the servers' listener, but I want all connections. There HAS to be a method I am missing somewhere to return all connections to a server or manager Please help!!

  Read the article

• Python - How to wake up a sleeping process- multiprocessing?

  - by user1162512

  I need to wake up a sleeping process ? The time (t) for which it sleeps is calculated as t = D/S . Now since s is varying, can increase or decrease, I need to increase/decrease the sleeping time as well. The speed is received over a UDP procotol. So, how do I change the sleeping time of a process, keeping in mind the following:- If as per the previous speed `S1`, the time to sleep is `(D/S1)` . Now the speed is changed, it should now sleep for the new time,ie (D/S2). Since, it has already slept for D/S1 time, now it should sleep for D/S2 - D/S1. How would I do it? As of right now, I'm just assuming that the speed will remain constant all throughout the program, hence not notifying the process. But how would I do that according to the above condition? def process2(): p = multiprocessing.current_process() time.sleep(secs1) # send some packet1 via UDP time.sleep(secs2) # send some packet2 via UDP time.sleep(secs3) # send some packet3 via UDP Also, as in threads, 1) threading.activeCount(): Returns the number of thread objects that are active. 2) threading.currentThread(): Returns the number of thread objects in the caller's thread control. 3) threading.enumerate(): Returns a list of all thread objects that are currently active. What are the similar functions for getting activecount, enumerate in multiprocessing?

  Read the article

• threading.Event wait function not signaled when subclassing Process class

  - by user1313404

  For following code never gets past the wait function in run. I'm certain I'm doing something ridiculously stupid, but since I'm not smart enough to figure out what, I'm asking. Any help is appreciated. Here is the code: import threading import multiprocessing from multiprocessing import Process class SomeClass(Process): def __init__(self): Process.__init__(self) self.event = threading.Event() self.event.clear() def continueExec(self): print multiprocessing.current_process().name print self print "Set:" + str(self.event.is_set()) self.event.set() print "Set:" + str(self.event.is_set()) def run(self): print "I'm running with it" print multiprocessing.current_process().name self.event.wait() print "I'm further than I was" print multiprocessing.current_process().name self.event.clear() def main(): s_list = [] for t in range(3): s = SomeClass() print "s:" + str(s) s_list.append(s) s.start() raw_input("Press enter to send signal") for t in range(3): print "s_list["+str(t)+"]:" + str(s_list[t]) s_list[t].continueExec() raw_input("Press enter to send signal") for t in range(3): s_list[t].join() print "All Done" if __name__ == "__main__": main()

  Read the article

• How to handle multi-processing of libraries which already spawn sub-processes?

  - by exhuma

  I am having some trouble coming up with a good solution to limit sub-processes in a script which uses a multi-processed library and the script itself is also multi-processed. Both, the library and script are modifiable by us. I believe the question is more about design than actual code, but for what it's worth, it's written in Python. The goal of the library is to hide implementation details of various internet routers. For that reason, the library has a "Proxy" factory method which takes the IP of a router as parameter. The factory then probes the device using a set of possible proxies. Usually, there is one proxy which immediately knows that is is able to send commands to this device. All others usually take some time to return (given a timeout). One thought was already to simply query the device for an identifier, and then select the proper proxy using that, but in order to do so, you would already need to know how to query the device. Abstracting this knowledge is one of the main purposes of the library, so that becomes a little bit of a "circular-requirement"/deadlock: To connect to a device, you need to know what proxy to use, and to know what proxy to create, you need to connect to a device. So probing the device is - as we can see - the best solution so far, apart from keeping a lookup-table somewhere. The library currently kills all remaining processes once a valid proxy has been found. And yes, there is always only one good proxy per device. Currently there are about 12 proxies. So if one create a proxy instance using the factory, 12 sub-processes are spawned. So far, this has been really useful and worked very well. But recently someone else wanted to use this library to "broadcast" a command to all devices. So he took the library, and wrote his own multi-processed script. This obviously spawned 12 * n processes where n is the number of IPs to which he broadcasted. This has given us two problems: The host on which the command was executed slowed down to a near halt. Aborting the script with CTRL+C ground the system to a total halt. Not even the hardware console responded anymore! This may be due to some Python strangeness which still needs to be investigated. Maybe related to http://bugs.python.org/issue8296 The big underlying question, is how to design a library which does multi-processing, so other applications which use this library and want to be multi-processed themselves do not run into system limitations. My first thought was to require a pool to be passed to the library, and execute all tasks in that pool. In that way, the person using the library has control over the usage of system resources. But my gut tells me that there must be a better solution. Disclaimer: My experience with multiprocessing is fairly limited. I have implemented a few straightforward which did not require access control to resources. So I have not yet any practical experience with semaphores or mutexes. p.s.: In the future, we may have enough information to do this without the probing. But the database which would contain the proper information is not yet operational. Also, the design about multiprocessing a multiprocessed library intrigues me :)

  Read the article

• Is there any kind of standard for 8086 multiprocessing?

  - by Earlz

  Back when I made an 8086 emulator I noticed that there was the LOCK prefix intended for synchonization in a multiprocessor environment. Yet the only multitasking I know of for the x86 arch. involves use of the APIC which didn't come around until either the Pentiums or 486s. Was there any kind of standard for 8086 multitasking or was it done by some manufacturer specific extensions to the instruction set and/or special ports? By standard, I mean things like: How do you separate the 2 processors if they both use the same memory? This is impossible without some kind of way to make each processor execute a different piece of code. (or cause an interrupt on only one processor)

  Read the article

1 2 3 4  | Next Page >