Search Results

Search found 15401 results on 617 pages for 'memory optimization'.

Page 156/617 | < Previous Page | 152 153 154 155 156 157 158 159 160 161 162 163  | Next Page >

• Memory efficient import many data files into panda DataFrame in Python

  - by richardh

  I import into a panda DataFrame a directory of |-delimited.dat files. The following code works, but I eventually run out of RAM with a MemoryError:. import pandas as pd import glob temp = [] dataDir = 'C:/users/richard/research/data/edgar/masterfiles' for dataFile in glob.glob(dataDir + '/master_*.dat'): print dataFile temp.append(pd.read_table(dataFile, delimiter='|', header=0)) masterAll = pd.concat(temp) Is there a more memory efficient approach? Or should I go whole hog to a database? (I will move to a database eventually, but I am baby stepping my move to pandas.) Thanks! FWIW, here is the head of an example .dat file: cik|cname|ftype|date|fileloc 1000032|BINCH JAMES G|4|2011-03-08|edgar/data/1000032/0001181431-11-016512.txt 1000045|NICHOLAS FINANCIAL INC|10-Q|2011-02-11|edgar/data/1000045/0001193125-11-031933.txt 1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-11|edgar/data/1000045/0001193125-11-005531.txt 1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-27|edgar/data/1000045/0001193125-11-015631.txt 1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2011-02-14|edgar/data/1000045/0000929638-11-00151.txt

  Read the article

• Allocate constant memory

  - by Vlad

  I'm trying to set my simulation params in constant memory but without luck (CUDA.NET). cudaMemcpyToSymbol function returns cudaErrorInvalidSymbol. The first parameter in cudaMemcpyToSymbol is string... Is it symbol name? actualy I don't understand how it could be resolved. Any help appreciated. //init, load .cubin float[] arr = new float[1]; arr[0] = 0.0f; int size = Marshal.SizeOf(arr[0]) * arr.Length; IntPtr ptr = Marshal.AllocHGlobal(size); Marshal.Copy(arr, 0, ptr, arr.Length); var error = CUDARuntime.cudaMemcpyToSymbol("param", ptr, 4, 0, cudaMemcpyKind.cudaMemcpyHostToDevice); my .cu file contain __constant__ float param;

  Read the article

• Which is the most memory leak safe approach.

  - by MattC

  I have a table of frequently updated information. This is presented using a container div with a div for each row, each row containing 10 divs. I am using setInterval to call a an asmx webservice that returns some json formatted information. On the success callback I call $("#myContainer").empty(); on the container div and recreate the rows and 10 nested divs for each row's columns. This page may be left to run for a whole day, so I am wary of updating the DOM like this as I have noticed that memory does rise for the browser over time (IE8). The other approach I am considering is to add an idea to the row div. When new results process each item of data, look for the corresponding row, if it exists overwrite the data in each div. If it doesn't exist (new data for example), append the row. What approaches have others used for this sort of long lived pseudo realtime information display. TIA

  Read the article

• How do you implement Software Transactional Memory?

  - by Joseph Garvin

  In terms of actual low level atomic instructions and memory fences (I assume they're used), how do you implement STM? The part that's mysterious to me is that given some arbitrary chunk of code, you need a way to go back afterward and determine if the values used in each step were valid. How do you do that, and how do you do it efficiently? This would also seem to suggest that just like any other 'locking' solution you want to keep your critical sections as small as possible (to decrease the probability of a conflict), am I right? Also, can STM simply detect "another thread entered this area while the computation was executing, therefore the computation is invalid" or can it actually detect whether clobbered values were used (and thus by luck sometimes two threads may execute the same critical section simultaneously without need for rollback)?

  Read the article

• Python In-memory table

  - by nsharish

  What is the right way to forming in-memory table in python with direct lookups for rows and columns.I thought of using dict of dicts this way, class Table(dict): def __getitem__(self, key): if key not in self: self[key]={} return dict.__getitem__(self, key) table = Table() table['row1']['column1'] = 'value11' table['row1']['column2'] = 'value12' table['row2']['column1'] = 'value21' table['row2']['column2'] = 'value22' >>>table {'row1':{'column1':'value11','column2':'value12'},'row2':{'column1':'value21','column2':'value22'}} I had difficulty in looking up for values in columns. >>>'row1' in table True >>>'value11' in table['row1'].values() True Now how do I do lookup if 'column1' has 'value11' Is this method of forming tables wrong?Is there a better way to implement such tables with easier lookups?.Thanks

  Read the article

• Out of Memory When Loading Java Entities

  - by Hugh Buchanan

  I have a terrible problem that hopefully has a very simple answer. I am running out of memory when I perform a basic If I have code like this: MyEntity myEntity; for (Object id: someIdList) { myEntity = find(id); // do something basic with myEntity } And the find() method is a standard EntityManager related method: public MyEntity find(Object id) { return em.find(mycorp.ejb.entity.MyEntity.class, id); } This code worked a couple of weeks ago, and works fine if there are fewer items in the database. The resulting error I am facing is: java.lang.OutOfMemoryError: GC overhead limit exceeded The exception is coming from oracle toplink calling some oracle jdbc methods. The loop exists because an EJBQL such as "select object(o) from MyEntity as o" will overload the application server when there are lots of records.

  Read the article

• phpDocumentor alternative consuming less memory

  - by Michal Cihar

  Okay, I'm fed up with phpDocumentator. It consumes way much more memory and time than I'm willing to give it. Does there exist some really compatible program to generate documentation for PHP code? I've tried PHPDoctor, which looks nice, but it has some missing features. I've also tried PhpDocGen, but it just bails out with some Perl errors, which I'm too lazy to study. Doxygen also does not seem to play well with our sources. PS: The documentation would be for phpMyAdmin, a little bit outdated documentation is here.

  Read the article

• Permutations of Varying Size

  - by waiwai933

  I'm trying to write a function in PHP that gets all permutations of all possible sizes. I think an example would be the best way to start off: $my_array = array(1,1,2,3); Possible permutations of varying size: 1 1 // * See Note 2 3 1,1 1,2 1,3 // And so forth, for all the sets of size 2 1,1,2 1,1,3 1,2,1 // And so forth, for all the sets of size 3 1,1,2,3 1,1,3,2 // And so forth, for all the sets of size 4 Note: I don't care if there's a duplicate or not. For the purposes of this example, all future duplicates have been omitted. What I have so far in PHP: function getPermutations($my_array){ $permutation_length = 1; $keep_going = true; while($keep_going){ while($there_are_still_permutations_with_this_length){ // Generate the next permutation and return it into an array // Of course, the actual important part of the code is what I'm having trouble with. } $permutation_length++; if($permutation_length>count($my_array)){ $keep_going = false; } else{ $keep_going = true; } } return $return_array; } The closest thing I can think of is shuffling the array, picking the first n elements, seeing if it's already in the results array, and if it's not, add it in, and then stop when there are mathematically no more possible permutations for that length. But it's ugly and resource-inefficient. Any pseudocode algorithms would be greatly appreciated. Also, for super-duper (worthless) bonus points, is there a way to get just 1 permutation with the function but make it so that it doesn't have to recalculate all previous permutations to get the next? For example, I pass it a parameter 3, which means it's already done 3 permutations, and it just generates number 4 without redoing the previous 3? (Passing it the parameter is not necessary, it could keep track in a global or static). The reason I ask this is because as the array grows, so does the number of possible combinations. Suffice it to say that one small data set with only a dozen elements grows quickly into the trillions of possible combinations and I don't want to task PHP with holding trillions of permutations in its memory at once.

  Read the article

• Logging Mechanism using memory mapping technique

  - by Tushar

  Just create a mapping of the file of the required size (CreateFileMapping or mmap), write the lines in the buffer and start over when the maximum number is reached. -- Your answer for write-a-circular-file-in-c. I am also writing the LogWriter module. In this caase i am mapping the whole file to the memory using mmap(). I am maintaining the Read and Write pointers.I want to write the log to the file in append mode. Then when logger service is started first time it writes it appends the logs. But when system gets shutdown next time when i run the service it doesn't append the data at the end. I want to maintain the write and read offsets even if system shut down.How to achieve this ..? How to find the how much data is written to the log file. ??

  Read the article

• How to optimize my PostgreSQL DB for prefix search?

  - by asmaier

  I have a table called "nodes" with roughly 1.7 million rows in my PostgreSQL db =#\d nodes Table "public.nodes" Column | Type | Modifiers --------+------------------------+----------- id | integer | not null title | character varying(256) | score | double precision | Indexes: "nodes_pkey" PRIMARY KEY, btree (id) I want to use information from that table for autocompletion of a search field, showing the user a list of the ten titles having the highest score fitting to his input. So I used this query (here searching for all titles starting with "s") =# explain analyze select title,score from nodes where title ilike 's%' order by score desc; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------- Sort (cost=64177.92..64581.38 rows=161385 width=25) (actual time=4930.334..5047.321 rows=161264 loops=1) Sort Key: score Sort Method: external merge Disk: 5712kB -> Seq Scan on nodes (cost=0.00..46630.50 rows=161385 width=25) (actual time=0.611..4464.413 rows=161264 loops=1) Filter: ((title)::text ~~* 's%'::text) Total runtime: 5260.791 ms (6 rows) This was much to slow for using it with autocomplete. With some information from Using PostgreSQL in Web 2.0 Applications I was able to improve that with a special index =# create index title_idx on nodes using btree(lower(title) text_pattern_ops); =# explain analyze select title,score from nodes where lower(title) like lower('s%') order by score desc limit 10; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------ Limit (cost=18122.41..18122.43 rows=10 width=25) (actual time=1324.703..1324.708 rows=10 loops=1) -> Sort (cost=18122.41..18144.60 rows=8876 width=25) (actual time=1324.700..1324.702 rows=10 loops=1) Sort Key: score Sort Method: top-N heapsort Memory: 17kB -> Bitmap Heap Scan on nodes (cost=243.53..17930.60 rows=8876 width=25) (actual time=96.124..1227.203 rows=161264 loops=1) Filter: (lower((title)::text) ~~ 's%'::text) -> Bitmap Index Scan on title_idx (cost=0.00..241.31 rows=8876 width=0) (actual time=90.059..90.059 rows=161264 loops=1) Index Cond: ((lower((title)::text) ~>=~ 's'::text) AND (lower((title)::text) ~<~ 't'::text)) Total runtime: 1325.085 ms (9 rows) So this gave me a speedup of factor 4. But can this be further improved? What if I want to use '%s%' instead of 's%'? Do I have any chance of getting a decent performance with PostgreSQL in that case, too? Or should I better try a different solution (Lucene?, Sphinx?) for implementing my autocomplete feature?

  Read the article

• remove ViewController from memory

  - by user262325

  hello everyone I hope to load an ViewController and do something then unload it from memory. if (self.vViewController5.view.superview==nil) { ViewController5 *blueController = [[ViewController5 alloc] initWithNibName:@"View5" bundle:nil]; self.vViewController5 = blueController; [self.vViewController5 setDelegate:self]; [blueController release]; } [self presentModalViewController:vViewController5 animated:YES]; later, call [self dismissModalViewControllerAnimated:YES]; but I found that dismissModalViewControllerAnimated does not trigger the event viewDidUnload of Viewcontroller5. I try function release but it caused program collapse. I also try removeFromSuperView but it does not trigger the event ViewDidUnload neither. Welcome any comment Thanks interdev

  Read the article

• Bitmapdata heavy usage - memory disaster (spark/FB4)

  - by keyle

  I've got a flex component which works pretty well but unfortunately turns into a disaster once used in a datagroup item renderer of about 40-50 items. Essentially it uses bitmapdata to take screenshot of a fully-rendered webpage in mx:HTML (this version of webkit rocks btw, miles better than flex 3). The code is pretty self-explanatory I think. http://noben.org/show/PageGrabber.mxml I've optimized it all I could, browsed, search for answers and already trimmed it down a lot, I'm desparate to reduce the memory usage (about 600mb after 100 draw) The Garbage collector has little effect. Thanks! Nic

  Read the article

• Out of Memory on Update or Delete of Service Reference

  - by Kelly

  I have a Service Reference for a WCF project that has just over a hundred endpoints in my ServiceHost web.config. Every time I attempt to update or delete the Service Reference, it fails with an out of memory exception. I am running Vista Ultimate SP2 64-bit with 8GB RAM. I can work around it by going outside the project and deleting the Service References folder, then coming back in and re-adding the Reference. Is this the only workaround that you know of? Thanks!

  Read the article

• Function-Local Static Const variable Initialization semantics.

  - by Hassan Syed

  The questions are in bold, for those that cannot be bothered reading a question in depth. This is a followup to this question. It is to do with the initialization semantics of static variables in functions. Static variables should be initialized once, and their internal state might be altered later - as I (currently) do in the linked question. However, the code in question does not require the feature to change the state of the variable later. Let me clarrify my position, since I don't require the string object's internal state to change. The code is for a trait class for meta programming, and as such would would benifit from a const char * const ptr -- thus Ideally a local cost static const variable is needed. My educated guess is that in this case the string in question will be optimally placed in memory by the link-loader, and that the code is more secure and maps to the intended semantics. This leads to the semantics of such a variable "The C++ Programming language Third Edition -- Stroustrup" does not have anything (that I could find) to say about this matter. All that is said is that the variable is initialized once when the flow of control of the thread first reaches the code. This leads me to ponder if the following code would be sensible, and if not what are the intended semantics ?. #include <iostream> const char * const GetString(const char * x_in) { static const char * const x = x_in; return x; } int main() { const char * const temp = GetString("yahoo"); std::cout << temp << std::endl; const char * const temp2 = GetString("yahoo2"); std::cout << temp2 << std::endl; } The following compiles on GCC and prints "yahoo" twice. Which is what I want -- However it might not be standards compliant (which is why I post this question). It might be more elegant to have two functions, "SetString" and "String" where the latter forwards to the first. If it is standards compliant does someone know of a templates implementation in boost (or elsewhere) ?

  Read the article

• Write wave files to memory in Java

  - by Cliff

  I'm trying to figure out why my servlet code creates wave files with improper headers. I use: AudioSystem.write( new AudioInputStream( new ByteArrayInputStream(memoryBytes), new AudioFormat(22000, 16, 1, true,false), memoryBytes.length ), AudioFileFormat.Type.WAVE, servletOutputStream ); taking a byte array from memory containing raw PCM samples and a servlet output stream that gets returned to the client. In the result I get a normal wave file but with zeros in the chunk size fields. Is the API broken? I would think that the size could be filled in using the size passed in the audio input stream. But now, after typing this out I'm thinking its not making this info available to the outer write() method on AudioSystem. It seems like the AudioSystem.write call needs a size parameter unless it is able to pull the size from the stream... which wouldn't work with an arbitrary sized stream. Does anyone know how to make this example work?

  Read the article

• HSQLDB Constraint Violation & SQL Query Log for an HSQLDB in-memory setup

  - by shipmaster

  We have a setup where we are using an embedded HSQLDB for backing Hibernate/JPA unit tests in java, and we are using the in-memory database mode since we simply want the database thrown away after the test run. My problem is that one of the tests is failing due to a constraint violation and HSQLDB lists the column as SYS_CT_286, and the query that appears in the log is the prepared statement where I cant see what the actual parameter values are (they are replaced by '?'). My questions are: 1- Is there a way in which I can see the actual SQL being executed? (like the mysql query log for example?). 2- What exactly is SYS_CT_286? it is not one of my columns, is it a generated column? is there something obvious that may be wrong with it? Thanks.

  Read the article

• Create an assembly in memory

  - by Jared I

  I'd like to create an assembly in memory, using an using the classes in Reflection.Emit Currently, I can create the assembly and get it's bytes using something like AssemblyBuilder builder = AppDomain.CurrentDomain.DefineDynamicAssembly(..., AssemblyBuilderAccess.Save); ... create the assembly ... builder.Save(targetFileName); using(FileStream fs = File.Open(targetFileName, FileMode.Open)) { ... read the bytes from the file stream ... } However, it does so by creating a file on the local filesystem. I don't actually need the file, just the bytes that would be in the file. Is it possible to create the assembly without writing any files?

  Read the article

• AVAudioPlayer Memory Leak - Media Player Framework

  - by Krishnan

  Hi Friends, I am using AVAudioPlayer object to play an audio. I create an audioPlayer object initially. I play an animation and when ever animation starts I play the audio and pause the audio when the animation is finished. I initially found three memory Leaks using Instruments. (The responsible caller mentioned was RegisterEmbedCodecs). After suggestion from a "ahmet emrah" in this forum to add MediaPlayer framework, the number of leaks reduced to one. And is there any way to completely get rid of it? Thanks and regards, krishnan.

  Read the article

• What are the pros and cons of using an in memory DB rather than a ThreadLocal

  - by Pangea

  We have been using ThreadLocal so far to carry some data so as to not clutter the API. However below are some of issues of using thread local that which I don't like: 1) Over the years the data items being carried in thread local has increased 2) Since we started using threads (for some light weight processing), we have also migrating these data to the threads in the pool and copying them back again I am thinking of using an in memory DB for these (we doesn't want to add this to the API). I wondering if this approach is good. What are the pros and cons? thx in advance.

  Read the article

• realloc()ing memory for a buffer used in recv()

  - by Hristo

  I need to recv() data from a socket and store it into a buffer, but I need to make sure get all of the data so I have things in a loop. So to makes sure I don't run out of room in my buffer, I'm trying to use realloc to resize the memory allocated to the buffer. So far I have: // receive response int i = 0; int amntRecvd = 0; char *pageContentBuffer = (char*) malloc(4096 * sizeof(char)); while ((amntRecvd = recv(proxySocketFD, pageContentBuffer + i, 4096, 0)) > 0) { i += amntRecvd; realloc(pageContentBuffer, 4096 + sizeof(pageContentBuffer)); } However, this doesn't seem to be working properly since Valgrind is complaining "valgrind: the 'impossible' happened:". Any advice as to how this should be done properly? Thanks, Hristo

  Read the article

• Memcache won't flush or clear memory

  - by pedalpete

  I've been trying to clear my memcache as I'm noticing the storage taking up almost 30% of server memory when using ps -aux. So I ran the following php code. $memcache = new Memcache; $memcache-connect("localhost",11211); $memcache-flush(); print_r($memcache-getStats()); This results in the output of ( [pid] = 4936 [uptime] = 27318915 [time] = 1255318611 [version] = 1.2.2 [pointer_size] = 64 [rusage_user] = 9.659531 [rusage_system] = 49.770433 [curr_items] = 57864 [total_items] = 128246 [bytes] = 1931734247 [curr_connections] = 1 [total_connections] = 128488 [connection_structures] = 17 [cmd_get] = 170288 [cmd_set] = 128246 [get_hits] = 45464 [get_misses] = 124824 [evictions] = 1009 [bytes_read] = 5607431213 [bytes_written] = 1806543589 [limit_maxbytes] = 2147483648 [threads] = 1 ) This should be fairly basic, but clearly, I'm missing something.

  Read the article

• XmlDocument caching memory usage

  - by mdsharpe

  We are seeing very high memory usage in .NET web applications which use XmlDocument. A small (~5MB) XML document is loaded into an XmlDocument object and stored in HttpContext.Cache for easy querying and XSLT transformation on each page load. The XML is modified on disk periodically so a cache has a dependency on the file. Such an application appears to be using hundreds of megabytes of RAM. I have experimented with requesting garbage collection on each request start, and this keeps the RAM usage far lower but I cannot imagine this is good practise. Does anyone have any suggestions as to how we can achieve the same goal but with lower RAM usage?

  Read the article

• NSDate & Memory management

  - by iFloh

  Hi, memory management still gives me grief. This time it is an NSDate iVar that I init using NSDate *myNSDate = [[NSDate date] firstDayOfMonth]; with a method call to - (NSDate *)firstDayOfMonth { NSDateComponents *tmpDateComponents = [[NSCalendar currentCalendar] components:NSYearCalendarUnit | NSMonthCalendarUnit | NSEraCalendarUnit | NSWeekCalendarUnit | NSWeekdayOrdinalCalendarUnit fromDate:self]; [tmpDateComponents setDay:1]; [tmpDateComponents setHour:0]; [tmpDateComponents setMinute:0]; [tmpDateComponents setSecond:0]; return [[NSCalendar currentCalendar] dateFromComponents:tmpDateComponents]; } At the end of the init call the retain count is at 1 (Note the iVar is not defined as a property). When I step into the viewWillAppear method the myNSDate has vanished. I tried to do an explicit retain on it, but that only lasts until I update the iVar using the above method again. I though - ok - I add the retain to the return of the function, but that makes the leak analyser throw up an error. What am I doing wrong?

  Read the article

• Struts/JSP/J2EE performance and memory profiling and issues

  - by Berlin Brown

  We are using Struts and having performance issues. And making heavy use of jsp includes, tiles, EL expressions. I am sure this is eating up a lot of memory and processing time. What are some approaches to profile the JSP page? What tools could I use? What should I look for when profiling? I have seen the code generated JSP Java Servlet Code and I see the bottlenecks but would rather measure it more accurately. This is under JDK1.5 and IBM Websphere 6.1 (RAD7)

  Read the article

• Memory section handling error

  - by Basilevs

  I'm getting a link time error: WARNING: /home/gulevich/development/camac-fedorov/camac/linux/k0607-lsi6/camac-k0607-lsi6.o (.ctors): unexpected non-allocatable section. Did you forget to use "ax"/"aw" in a .S file? Note that for example <linux/init.h> contains section definitions for use in .S files. The code causing the error (assembly in C source): # if defined(__ELF__) # define __SECTION_FLAGS ", \"aw\" , @progbits" /* writable flag needed for ld ".[cd]tors" sections bug workaround) */ # elif defined(__COFF__) # define __SECTION_FLAGS ", \"dr\"" /* untested, may be writable flag needed */ # endif asm ( ".section .ctors" __SECTION_FLAGS "\n" ".globl __ctors_begin__\n" "__ctors_begin__:\n" ".previous\n" ); Is there any way to fix this? The idea is to put a varaible __ctors_begin__ at the beginning of a certain memory section. This code is a legacy that worked fine using a different build system and older compiler. Meaning of this assembly code explained in an answer to my previous question.

  Read the article

< Previous Page | 152 153 154 155 156 157 158 159 160 161 162 163  | Next Page >