Search Results

Search found 7077 results on 284 pages for 'concurrent processing'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 138/284 | < Previous Page | 134 135 136 137 138 139 140 141 142 143 144 145  | Next Page >

  • javax.servlet.ServletException - how could i get to the cause?

    - by Michal
    Hi, i'm getting a very strange error while opening one of the pages in my web app. The application is built on Seam 2.2 and is using JSF (RichFaces) as a view technology. I run it on Tomcat 6. The error i'm describing doesn't occur on my machine (Mac OS X), but it does on my client's dev machines (Windows) and on the server (Linux Debian). I'm sure i'm running the same version on each system and i have tried connecting to the same database... In logs everything looks fine - each next JSF Phase executes normally, and after the last one, there is this moment when the request starts processing for the SEAM debug page... And this is the stack trace i see on the debug page (nothing is logged): Exception during request processing: Caused by javax.servlet.ServletException with message: "Servlet execution threw an exception" org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:313) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:388) org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:515) org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.HotDeployFilter.doFilter(HotDeployFilter.java:53) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) pl.mgibowski.alterium.util.LoggingFilter.doFilter(LoggingFilter.java:18) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) java.lang.Thread.run(Thread.java:619) Exception without any cause... I was trying to catch the exception with my custom Filter (LoggingFilter.java that you can see on the strack trace), using this code: try { filterChain.doFilter(servletRequest, servletResponse); } catch (Throwable e) { e.printStackTrace(); System.out.println("Stack trace:"); System.out.println(e.getStackTrace()); System.out.println("Cause:"); System.out.println(e.getCause()); } But it doesn't catch anything, the line 18 from the stack trace is this one: filterChain.doFilter(servletRequest, servletResponse); nothing gets caught by the try block. Anybody has any ideas about how could i get closer to the real cause?

    Read the article

  • Offloading to HLSL/GPU without displaying?

    - by George R
    As far as I know, certain mathematical functions like FFTs and perlin noise, etc. can be much faster when done on the GPU as a pixel shader. My question is, if I wanted to exploit this to calculate results and stream to bitmaps, could I do it without needing to actually display it in Silverlight or something? More specifically, I was thinking of using this for large terrain generation involving lots of perlin and other noises, and post-processing like high passes and deriving normals from heightmaps, etc, etc.

    Read the article

  • Altering the ASP.NET MVC 2 ActionResult on HTTP post

    - by Inge Henriksen
    I want to do some processing on a attribute before returning the view. If I set the appModel.Markup returned in the HttpPost ActionResult method below to "modified" it still says "original" on the form. Why cant I modify my attribute in a HttpGet ActionResult method? [HttpGet] public ActionResult Index() { return View(new MyModel { Markup = "original" }); } [HttpPost] public ActionResult Index(MyModel appModel) { return View(new MyModel { Markup = "modified" }); }

    Read the article

  • What’s the Minimum System Spec Recommended For Developer Laptop

    - by DaveDev
    I'll be regularly running Visual Studio 2010 Professional, SQLServer Express, Office and at least 1 virtual environment running a Linux Distro. I want the machine to be snappy and responsive even when doing a reasonable amount of Development work. I want to spend what it takes for this, but I don't want to go overboard spending more than I need to. I won't be playing many games or graphics processing so i won't need a monster of a machine. Any recommendations?

    Read the article

  • Why use Monte-Carlo method?

    - by Gili
    When should the Monte-Carlo method be used? For example, why did Joel decide to use the Monte-Carlo method for Evidence Based Scheduling instead of methodically processing all user data for the past year?

    Read the article

  • Http Request Monitoring Tool

    - by noli
    Hi! I would like to know if anybody can recommend an Http Request Monitoring Tool aside from HttpWatch and Firebug. What I want from the tool is for it to show me the time it took the request to arrive at the web server. HttpWatch can show me the network latency and the server times in one result but i want them separately. My goal is to isolate the network latency from the server processing times. Thanks.

    Read the article

  • Umbraco CMS stripping ALT tags from images when content saved

    - by Yucel
    Umbraco CMS stripping ALT tags from images when content saved Umbraco is taking this < img alt="Your Title - for example Mr., Mrs., Ms." src="../media/21283/q16x16.gif" width="16" height="15" /> and turning it into this. <img alt="" src="/media/21283/q16x16.gif" width="16" height="15" rel="16,15" /> If I alter the alt tag after this processing then the alt tag is saved.

    Read the article

  • BasicHTTPServer, SimpleHTTPServer and concurrency

    - by braindump
    I'm writing a small web server for testing purposes using python, BasicHTTPServer and SimpleHTTPServer. It looks like it's processing one request at a time. Is there any way to make it a little faster without messing around too deeply? Basicly my code looks as the following and I'd like to keep it this simple ;) os.chdir(webroot) httpd = BaseHTTPServer.HTTPServer(("", port), SimpleHTTPServer.SimpleHTTPRequestHandler) print("Serving directory %s on port %i" %(webroot, port) ) try: httpd.serve_forever() except KeyboardInterrupt: print("Server stopped.")

    Read the article

  • any good free C DSP library?

    - by Juan
    Hi everybody I am developing an application to process geophysical signals; Right now I have done everything in octave and its digital signal processing toolbox, speed is not bad, however the application specifications say I need to port to the final algorithm to C; I am doing lots of filtering, re-sampling and signal manipulation/characterization with FFTs and cepstrums. do you know a good free C library for DSP packaged with filter design, resampling, fft, etc? Thanks a lot for any suggestion

    Read the article

  • django internationalization doesn't work

    - by xRobot
    I have: created translation strings in the template and in the application view. run this command: django-admin.py makemessages -l it and the file it/LC_MESSAGES/django.po has been created run this command: django-admin.py compilemessages and I receive: processing file django.po in /home/jobber/Desktop/library/books/locale/it/LC_MESSAGES set the language code in settings.py: LANGUAGE_CODE = 'it-IT' but.... translation doesn't work !! I always see english text. Why ?

    Read the article

  • java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory

    - by Sanjeev
    I am using My-eclipse and doing a struts project there is no syntax error but on starting tomcat server the following error appear in console. java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory and javax.servlet.UnavailableException: Parsing error processing resource path jndi:/localhost/strutspro/WEB-INF/struts-config.xml any idea whats the problem.

    Read the article

  • Is there any performance comparison between Perl web frameworks?

    - by DVK
    I have seen mentions (which sounded like unsubstantiated opinions, and dated ones at that) that Embperl is the fastest Perl web framework. I was wondering if there's a consensus on the relative speed of the major stable Perl web frameworks, or ideally, some sort of fact-based performance comparisons between implementations of the same sample webapps, or individual functionalities (e.g. session handling or form data processing), etc...?

    Read the article

  • Enable iPhone accelerometer while screen is locked..

    - by kamziro
    So apparently it is possible to keep the processor going processing stuff while the screen is locked, as indicated here: http://stackoverflow.com/questions/1551712/running-iphone-apps-while-in-sleep-mode However, after testing with the example code, UIAccelerometer will just stop giving value as soon as the device is locked pronto. Is there a way to force otherwise?

    Read the article

  • Any ideas on a WCF Interceptor for a REST WebGet request to change the Url/Querystring parameters

    - by REA_ANDREW
    Does any one know of a way I can intercept a REST Get request inside WCF, so for example I could change the value of any of the querystring parameters. So I need to have some code run, prior to WCF Evaluating the UriTemplate of the WebGet attribute and be able to edit it, before returning the value which it will use to continue processing the request. Any help would be greatly appreciated Andrew

    Read the article

  • Business Intelligence Development Studio encounter error on Rendering

    - by user366796
    I try to develop a data processing extension for SSRS 2008 in order to access database through an entity framework. But when I copy and register the extension in BI Development Studio, it gives me an error message while loading the extension". I built it with targeting framework 4.0 by using Visual Studio 2010 because my data model class library was built with that version of target framework. Any help will be greatly appreciated. Thanks.

    Read the article

  • Fast file search algorithm for IP addresses

    - by Dave Jarvis
    Question What is the fastest way to find if an IP address exists in a file that contains IP addresses sorted as: 219.93.88.62 219.94.181.87 219.94.193.96 220.1.72.201 220.110.162.50 220.126.52.187 220.126.52.247 Constraints No database (e.g., MySQL, PostgreSQL, Oracle, etc.). Infrequent pre-processing is allowed (see possibilities section) Would be nice not to have to load the file each query (131Kb) Uses under 5 megabytes of disk space File Details One IP address per line 9500+ lines Possible Solutions Create a directory hierarchy (radix tree?) then use is_dir() (sadly, this uses 87 megabytes)

    Read the article

  • Messagebox in ASP.NET 3.5 with AJAX

    - by Jim Beam
    I have a ASP.NET 3.5 web site with an AJAX update panel. I simply need to process some server side code and then issue a user prompt that says "Code processing complete". I know there is supposed to be support for Msgbox-esque methods in ASP.NET but I can't find them and any other JavaScript based solutions don't work effectively when you have an update panel. Help.

    Read the article

  • django: search forms and redirect

    - by gruszczy
    After processing form from POST I should redirect, to prevent user from hitting back. However, I am using form to determine search query on a database, so I need to either pass params to the redirected site or the result of a search. Or maybe there is some other good practice, how to solve this problem? Maybe in this situation I am allowed not to redirect (nothing happens, if user performs search again).

    Read the article

  • What is the fastest cyclic synchronization in Java (ExecutorService vs. CyclicBarrier vs. X)?

    - by Alex Dunlop
    Which Java synchronization construct is likely to provide the best performance for a concurrent, iterative processing scenario with a fixed number of threads like the one outlined below? After experimenting on my own for a while (using ExecutorService and CyclicBarrier) and being somewhat surprised by the results, I would be grateful for some expert advice and maybe some new ideas. Existing questions here do not seem to focus primarily on performance, hence this new one. Thanks in advance! The core of the app is a simple iterative data processing algorithm, parallelized to the spread the computational load across 8 cores on a Mac Pro, running OS X 10.6 and Java 1.6.0_07. The data to be processed is split into 8 blocks and each block is fed to a Runnable to be executed by one of a fixed number of threads. Parallelizing the algorithm was fairly straightforward, and it functionally works as desired, but its performance is not yet what I think it could be. The app seems to spend a lot of time in system calls synchronizing, so after some profiling I wonder whether I selected the most appropriate synchronization mechanism(s). A key requirement of the algorithm is that it needs to proceed in stages, so the threads need to sync up at the end of each stage. The main thread prepares the work (very low overhead), passes it to the threads, lets them work on it, then proceeds when all threads are done, rearranges the work (again very low overhead) and repeats the cycle. The machine is dedicated to this task, Garbage Collection is minimized by using per-thread pools of pre-allocated items, and the number of threads can be fixed (no incoming requests or the like, just one thread per CPU core). V1 - ExecutorService My first implementation used an ExecutorService with 8 worker threads. The program creates 8 tasks holding the work and then lets them work on it, roughly like this: // create one thread per CPU executorService = Executors.newFixedThreadPool( 8 ); ... // now process data in cycles while( ...) { // package data into 8 work items ... // create one Callable task per work item ... // submit the Callables to the worker threads executorService.invokeAll( taskList ); } This works well functionally (it does what it should), and for very large work items indeed all 8 CPUs become highly loaded, as much as the processing algorithm would be expected to allow (some work items will finish faster than others, then idle). However, as the work items become smaller (and this is not really under the program's control), the user CPU load shrinks dramatically: blocksize | system | user | cycles/sec 256k 1.8% 85% 1.30 64k 2.5% 77% 5.6 16k 4% 64% 22.5 4096 8% 56% 86 1024 13% 38% 227 256 17% 19% 420 64 19% 17% 948 16 19% 13% 1626 Legend: - block size = size of the work item (= computational steps) - system = system load, as shown in OS X Activity Monitor (red bar) - user = user load, as shown in OS X Activity Monitor (green bar) - cycles/sec = iterations through the main while loop, more is better The primary area of concern here is the high percentage of time spent in the system, which appears to be driven by thread synchronization calls. As expected, for smaller work items, ExecutorService.invokeAll() will require relatively more effort to sync up the threads versus the amount of work being performed in each thread. But since ExecutorService is more generic than it would need to be for this use case (it can queue tasks for threads if there are more tasks than cores), I though maybe there would be a leaner synchronization construct. V2 - CyclicBarrier The next implementation used a CyclicBarrier to sync up the threads before receiving work and after completing it, roughly as follows: main() { // create the barrier barrier = new CyclicBarrier( 8 + 1 ); // create Runable for thread, tell it about the barrier Runnable task = new WorkerThreadRunnable( barrier ); // start the threads for( int i = 0; i < 8; i++ ) { // create one thread per core new Thread( task ).start(); } while( ... ) { // tell threads about the work ... // N threads + this will call await(), then system proceeds barrier.await(); // ... now worker threads work on the work... // wait for worker threads to finish barrier.await(); } } class WorkerThreadRunnable implements Runnable { CyclicBarrier barrier; WorkerThreadRunnable( CyclicBarrier barrier ) { this.barrier = barrier; } public void run() { while( true ) { // wait for work barrier.await(); // do the work ... // wait for everyone else to finish barrier.await(); } } } Again, this works well functionally (it does what it should), and for very large work items indeed all 8 CPUs become highly loaded, as before. However, as the work items become smaller, the load still shrinks dramatically: blocksize | system | user | cycles/sec 256k 1.9% 85% 1.30 64k 2.7% 78% 6.1 16k 5.5% 52% 25 4096 9% 29% 64 1024 11% 15% 117 256 12% 8% 169 64 12% 6.5% 285 16 12% 6% 377 For large work items, synchronization is negligible and the performance is identical to V1. But unexpectedly, the results of the (highly specialized) CyclicBarrier seem MUCH WORSE than those for the (generic) ExecutorService: throughput (cycles/sec) is only about 1/4th of V1. A preliminary conclusion would be that even though this seems to be the advertised ideal use case for CyclicBarrier, it performs much worse than the generic ExecutorService. V3 - Wait/Notify + CyclicBarrier It seemed worth a try to replace the first cyclic barrier await() with a simple wait/notify mechanism: main() { // create the barrier // create Runable for thread, tell it about the barrier // start the threads while( ... ) { // tell threads about the work // for each: workerThreadRunnable.setWorkItem( ... ); // ... now worker threads work on the work... // wait for worker threads to finish barrier.await(); } } class WorkerThreadRunnable implements Runnable { CyclicBarrier barrier; @NotNull volatile private Callable<Integer> workItem; WorkerThreadRunnable( CyclicBarrier barrier ) { this.barrier = barrier; this.workItem = NO_WORK; } final protected void setWorkItem( @NotNull final Callable<Integer> callable ) { synchronized( this ) { workItem = callable; notify(); } } public void run() { while( true ) { // wait for work while( true ) { synchronized( this ) { if( workItem != NO_WORK ) break; try { wait(); } catch( InterruptedException e ) { e.printStackTrace(); } } } // do the work ... // wait for everyone else to finish barrier.await(); } } } Again, this works well functionally (it does what it should). blocksize | system | user | cycles/sec 256k 1.9% 85% 1.30 64k 2.4% 80% 6.3 16k 4.6% 60% 30.1 4096 8.6% 41% 98.5 1024 12% 23% 202 256 14% 11.6% 299 64 14% 10.0% 518 16 14.8% 8.7% 679 The throughput for small work items is still much worse than that of the ExecutorService, but about 2x that of the CyclicBarrier. Eliminating one CyclicBarrier eliminates half of the gap. V4 - Busy wait instead of wait/notify Since this app is the primary one running on the system and the cores idle anyway if they're not busy with a work item, why not try a busy wait for work items in each thread, even if that spins the CPU needlessly. The worker thread code changes as follows: class WorkerThreadRunnable implements Runnable { // as before final protected void setWorkItem( @NotNull final Callable<Integer> callable ) { workItem = callable; } public void run() { while( true ) { // busy-wait for work while( true ) { if( workItem != NO_WORK ) break; } // do the work ... // wait for everyone else to finish barrier.await(); } } } Also works well functionally (it does what it should). blocksize | system | user | cycles/sec 256k 1.9% 85% 1.30 64k 2.2% 81% 6.3 16k 4.2% 62% 33 4096 7.5% 40% 107 1024 10.4% 23% 210 256 12.0% 12.0% 310 64 11.9% 10.2% 550 16 12.2% 8.6% 741 For small work items, this increases throughput by a further 10% over the CyclicBarrier + wait/notify variant, which is not insignificant. But it is still much lower-throughput than V1 with the ExecutorService. V5 - ? So what is the best synchronization mechanism for such a (presumably not uncommon) problem? I am weary of writing my own sync mechanism to completely replace ExecutorService (assuming that it is too generic and there has to be something that can still be taken out to make it more efficient). It is not my area of expertise and I'm concerned that I'd spend a lot of time debugging it (since I'm not even sure my wait/notify and busy wait variants are correct) for uncertain gain. Any advice would be greatly appreciated.

    Read the article

  • Solving embarassingly parallel problems using Python multiprocessing

    - by gotgenes
    How does one use multiprocessing to tackle embarrassingly parallel problems? Embarassingly parallel problems typically consist of three basic parts: Read input data (from a file, database, tcp connection, etc.). Run calculations on the input data, where each calculation is independent of any other calculation. Write results of calculations (to a file, database, tcp connection, etc.). We can parallelize the program in two dimensions: Part 2 can run on multiple cores, since each calculation is independent; order of processing doesn't matter. Each part can run independently. Part 1 can place data on an input queue, part 2 can pull data off the input queue and put results onto an output queue, and part 3 can pull results off the output queue and write them out. This seems a most basic pattern in concurrent programming, but I am still lost in trying to solve it, so let's write a canonical example to illustrate how this is done using multiprocessing. Here is the example problem: Given a CSV file with rows of integers as input, compute their sums. Separate the problem into three parts, which can all run in parallel: Process the input file into raw data (lists/iterables of integers) Calculate the sums of the data, in parallel Output the sums Below is traditional, single-process bound Python program which solves these three tasks: #!/usr/bin/env python # -*- coding: UTF-8 -*- # basicsums.py """A program that reads integer values from a CSV file and writes out their sums to another CSV file. """ import csv import optparse import sys def make_cli_parser(): """Make the command line interface parser.""" usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV", __doc__, """ ARGUMENTS: INPUT_CSV: an input CSV file with rows of numbers OUTPUT_CSV: an output file that will contain the sums\ """]) cli_parser = optparse.OptionParser(usage) return cli_parser def parse_input_csv(csvfile): """Parses the input CSV and yields tuples with the index of the row as the first element, and the integers of the row as the second element. The index is zero-index based. :Parameters: - `csvfile`: a `csv.reader` instance """ for i, row in enumerate(csvfile): row = [int(entry) for entry in row] yield i, row def sum_rows(rows): """Yields a tuple with the index of each input list of integers as the first element, and the sum of the list of integers as the second element. The index is zero-index based. :Parameters: - `rows`: an iterable of tuples, with the index of the original row as the first element, and a list of integers as the second element """ for i, row in rows: yield i, sum(row) def write_results(csvfile, results): """Writes a series of results to an outfile, where the first column is the index of the original row of data, and the second column is the result of the calculation. The index is zero-index based. :Parameters: - `csvfile`: a `csv.writer` instance to which to write results - `results`: an iterable of tuples, with the index (zero-based) of the original row as the first element, and the calculated result from that row as the second element """ for result_row in results: csvfile.writerow(result_row) def main(argv): cli_parser = make_cli_parser() opts, args = cli_parser.parse_args(argv) if len(args) != 2: cli_parser.error("Please provide an input file and output file.") infile = open(args[0]) in_csvfile = csv.reader(infile) outfile = open(args[1], 'w') out_csvfile = csv.writer(outfile) # gets an iterable of rows that's not yet evaluated input_rows = parse_input_csv(in_csvfile) # sends the rows iterable to sum_rows() for results iterable, but # still not evaluated result_rows = sum_rows(input_rows) # finally evaluation takes place as a chain in write_results() write_results(out_csvfile, result_rows) infile.close() outfile.close() if __name__ == '__main__': main(sys.argv[1:]) Let's take this program and rewrite it to use multiprocessing to parallelize the three parts outlined above. Below is a skeleton of this new, parallelized program, that needs to be fleshed out to address the parts in the comments: #!/usr/bin/env python # -*- coding: UTF-8 -*- # multiproc_sums.py """A program that reads integer values from a CSV file and writes out their sums to another CSV file, using multiple processes if desired. """ import csv import multiprocessing import optparse import sys NUM_PROCS = multiprocessing.cpu_count() def make_cli_parser(): """Make the command line interface parser.""" usage = "\n\n".join(["python %prog INPUT_CSV OUTPUT_CSV", __doc__, """ ARGUMENTS: INPUT_CSV: an input CSV file with rows of numbers OUTPUT_CSV: an output file that will contain the sums\ """]) cli_parser = optparse.OptionParser(usage) cli_parser.add_option('-n', '--numprocs', type='int', default=NUM_PROCS, help="Number of processes to launch [DEFAULT: %default]") return cli_parser def main(argv): cli_parser = make_cli_parser() opts, args = cli_parser.parse_args(argv) if len(args) != 2: cli_parser.error("Please provide an input file and output file.") infile = open(args[0]) in_csvfile = csv.reader(infile) outfile = open(args[1], 'w') out_csvfile = csv.writer(outfile) # Parse the input file and add the parsed data to a queue for # processing, possibly chunking to decrease communication between # processes. # Process the parsed data as soon as any (chunks) appear on the # queue, using as many processes as allotted by the user # (opts.numprocs); place results on a queue for output. # # Terminate processes when the parser stops putting data in the # input queue. # Write the results to disk as soon as they appear on the output # queue. # Ensure all child processes have terminated. # Clean up files. infile.close() outfile.close() if __name__ == '__main__': main(sys.argv[1:]) These pieces of code, as well as another piece of code that can generate example CSV files for testing purposes, can be found on github. I would appreciate any insight here as to how you concurrency gurus would approach this problem. Here are some questions I had when thinking about this problem. Bonus points for addressing any/all: Should I have child processes for reading in the data and placing it into the queue, or can the main process do this without blocking until all input is read? Likewise, should I have a child process for writing the results out from the processed queue, or can the main process do this without having to wait for all the results? Should I use a processes pool for the sum operations? If yes, what method do I call on the pool to get it to start processing the results coming into the input queue, without blocking the input and output processes, too? apply_async()? map_async()? imap()? imap_unordered()? Suppose we didn't need to siphon off the input and output queues as data entered them, but could wait until all input was parsed and all results were calculated (e.g., because we know all the input and output will fit in system memory). Should we change the algorithm in any way (e.g., not run any processes concurrently with I/O)?

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 134 135 136 137 138 139 140 141 142 143 144 145  | Next Page >