Search Results

Search found 12357 results on 495 pages for 'memory lane'.

Page 6/495 | < Previous Page | 2 3 4 5 6 7 8 9 10 11 12 13  | Next Page >

• High memory utilization by sqlservr.exe process

  - by abdul samad

  Sub:High memory utilization by sqlservr.exe process. When I look into task manager --processes or by using perfmon memory counters(Sqlserver:memory manager:Target server memory and Total server memory) I am getting high memory utilization by sqlservr.exe process nearly 8 GB (Target server memory counter) and 7.95 GB (Total server memory). and when I restart the MSSQLSERVER service it again shoots up to the same size. I am getting this issue quite frequently. Please help me out in identifying why sql server is using so much memory and how to find out what query , stored procedure etc is making sql server use that much memory. * I am not using any triggers or cursors in my code. Thanks

  Read the article

• Some Memory Slots Not Working on MSI FM2-A85XA-G65 Motherboard

  - by Mike Ciaraldi

  Short version of question: Does anyone have an MSI FM2-A85XA-G65 motherboard, who can confirm that all four memory slots work? Long version: Several months ago I bought an MSI FM2-A85XA-G65 motherboard at Newegg. At that point I installed an AMD A8-5500 processor and two sticks of Corsair Vengeance 8 GB DDR3-1866 memory, and put it into my file server. I installed the RAM in slots 1 and 3, as directed in the manual, to enable dual-channel memory access. It seemed to work fine, so I bought a second identical mobo (which arrived dead, but was quickly replaced by Newegg) and set of RAM, installed an A10-5800K, and put that into my production Linux machine. Again, it seemed to work well. Eventually I happened to notice that on the server only 8 GB of RAM appeared in the BIOS. I tried each of the slots and memory modules individually and in various combinations. I even swapped processors with the production machine. The result was that putting memory in slots 1 and 2 worked (showing a total of 16 GB), but any memory in slots 3 or 4 was not recognized. However, all four memory slots in the production machine worked, and I confirmed this with both processors. I contacted MSI and arranged to ship the defective mobo back to them for replacement under warranty. I did not want my file server to be down in the interim, and I had another machine I wanted to upgrade, so I bought a third identical mobo to use. That one had the same problem -- only memory slots 1 and 2 worked. I tested it thoroughly with multiple processors and memory sticks. I sent the defective mobo back to MSI and they sent me a new one. This has the same memory slot problem. So I sent it back. The replacement arrived the other day and shows the same problem. I contacted MSI yet again and they said that nobody else has reported memory slot problems on this board and it must be my processor. So my score so far is, out of six boards of this model, I have: One where all four slots work. One which was dead on arrival. Four where only memory slots 1 and 2 work. Before I tear my other machines apart and start swapping processors again I thought I would ask if anyone else has this exact model motherboard and could confirm that all four memory slots either do or do not work. According to MSI you should be able to just plug a single memory module into any of the slots and it will work (and it does on the one mobo I have which works correctly). If you have not yet used all four slots, this is a good time to test them so you know if you can expand your memory in the future. Thanks in advance to anyone who can help.

  Read the article

• Fastest Memory (within reason) for a MotherBoard [on hold]

  - by sampson

  I was wondering if it would be OK to use DDR3 3000 memory with Asus Maximus VI Impact MotherBoard, Intel® Core™ i3-4130T Processor and Steamcom's FC8 case The purpose of this machine is for a HTPC (Home Theater Personal Computer) system, only, no gaming. The case is fan less as is the CPU cooling system. Also, would it be worth it, heat wise, to go past the 1600 memory type? I mean, would DDR 3000 make the box that much faster to make it worthwhile? The Processor has a TDP rating of 35 W. The memory specifications for the processor are: Memory Specifications Max Memory Size (dependent on memory type) 32 GB Memory Types DDR3-1333/1600 # of Memory Channels 2 Max Memory Bandwidth 25.6 GB/s ECC Memory Supported ‡ Yes The FC8 case's heat displacement system is rated at 95 W TDP

  Read the article

• ruby/ruby on rails memory leak detection

  - by Josh Moore

  I wrote a small web app using ruby on rails, its main purpose is to upload, store, and display results from xml(files can be up to several MB) files. After running for about 2 months I noticed that the mongrel process was using about 4GB of memory. I did some research on debugging ruby memory leaks and could not find much. So I have two questions. Are there any good tools that can be used to find memory leaks in Ruby/rails? What type of coding patterns cause memory leaks in ruby?

  Read the article

• How important is managing memory in Objective-C?

  - by Alex Mcp

  Background: I'm (jumping on the bandwagon and) starting learning about iPhone/iPad development and Objective-C. I have a great background in web development and most of my programming is done in javascript (no libraries), Ruby, and PHP. Question: I'm learning about allocating and releasing memory in Objective-C, and I see it as quite a tricky task to layer on top of actually getting the farking thing to run. I'm trying to get a sense of applications that are out there and what will happen with a poorly memory-managed program. A) Are apps usually released with no memory leaks? Is this a feasible goal, or do people more realistically just excise the worst offenders and that's ok? B) If I make an NSString for a title of a view, let's say, and forget to deallocate it it, does this really only become a problem if I recreate that string repeatedly? I imagine what I'm doing is creating an overhead of the memory needed to store that string, so it's probably quite piddling (a few bytes?) However if I have a rapidly looping cycle in a game that 'leaks' an int every cycle or something, that would overflow the app quite quickly. Are these assumptions correct? Sorry if this isn't up the community-wiki alley, I'm just trying to get a handle on how to think about memory and how careful I'll need to be. Any anecdotes or App Store-submitted app experiences would be awesome to hear as well.

  Read the article

• Accessing outside the memory allocated by the program. (Accessing other app's memory)

  - by Auxiliary

  Is there a way to access (read or free) memory chunks that are outside the memory that is allocated for the program without getting access violation exceptions. Well what I actually would like to understand apart from this, is how a memory cleaner (system garbage collector) works. I've always wanted to write such a program. (The language isn't an issue) Thanks in advance :)

  Read the article

• Memory usage in Flash / Flex / AS3

  - by ggambett

  I'm having some trouble with memory management in a flash app. Memory usage grows quite a bit, and I've tracked it down to the way I load assets. I embed several raster images in a class Embedded, like this [Embed(source="/home/gabriel/text_hard.jpg")] public static var ASSET_text_hard_DOT_jpg : Class; I then instance the assets this way var pClass : Class = Embedded[sResource] as Class; return new pClass() as Bitmap; At this point, memory usage goes up, which is perfectly normal. However, nulling all the references to the object doesn't free the memory. Based on this behavior, looks like the flash player is creating an instance of the class the first time I request it, but never ever releases it - not without references, calling System.gc(), doing the double LocalConnection trick, or calling dispose() on the BitmapData objects. Of course, this is very undesirable - memory usage would grow until everything in the SWFs is instanced, regardless of whether I stopped using some asset long ago. Is my analysis correct? Can anything be done to fix this?

  Read the article

• How do I mock memory allocation failures ?

  - by Andrei Ciobanu

  I want to extensively test some pieces of C code for memory leaks. On my machine I have 4 Gb of RAM, so it's very unlikely for a dynamic memory allocation to fail. Still I want to see the comportment of the code if memory allocation fails, and see if the recover mechanism is "strong" enough. What do you suggest ? How do I emulate an environment with lower memory specs ? How do i mock my tests ? EDIT: I want my tests to be code independent. I only have "access" to return values for different functions in the library I am testing. I am not supposed to write "test logic" inside the code I am testing.

  Read the article

• Protecting an Application's Memory From Tampering

  - by Changeling

  We are adding AES 256 bit encryption to our server and client applications for encrypting the TCP/IP traffic containing sensitive information. We will be rotating the keys daily. Because of that, the keys will be stored in memory with the applications. Key distribution process: Each server and client will have a list of initial Key Encryption Key's (KEK) by day If the client has just started up or the server has just started up, the client will request the daily key from the server using the initial key. The server will respond with the daily key, encrypted with the initial key. The daily key is a randomly generated set of alphanumeric characters. We are using AES 256 bit encryption. All subsequent communications will be encrypted using that daily key. Nightly, the client will request the new daily key from the server using the current daily key as the current KEK. After the client gets the new key, the new daily key will replace the old daily key. Is it possible for another bad application to gain access to this memory illegally or is this protected in Windows? The key will not be written to a file, only stored in a variable in memory. If an application can access the memory illegally, how can you protect the memory from tampering? We are using C++ and XP (Vista/7 may be an option in the future so I don't know if that changes the answer).

  Read the article

• Memory Warning but Small Live Bytes

  - by Kamchatka

  Hi everyone, In my application, I get a memory warning of level 1 and then 2 after repeating some action (choosing a picture + processing) several times and then a crash. The leak tool doesn't show any leak. I'm also following the Allocations tool in Instruments and my Live Bytes are roughly 4 MB, overall I allocate 113 MB. At maximum I have maybe 20 MB in memory when the picture is loaded. Since I have to repeat an action to get to the crash, it is very likely to be a memory leak. However, I don't know how to locate it since my live bytes are 4 MB and things supposed to be allocated (apart a small leak of ~100 KB in the UIImagePickerController). How much can I trust the memory leak/allocation tools? Would you have an advice to help me locate the reason of the problem?

  Read the article

• Objective-c for the iphone: Mystery memory leak

  - by user200341

  My application seems to have 4 memory leaks (on the device, running instruments). The memory leaks seems to come from this code: NSURL *url = [self getUrl:destination]; [destination release]; NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:url]; [url release]; [request setHTTPMethod:@"GET"]; [request addValue:@"application/json" forHTTPHeaderField:@"content-type"]; NSURLConnection *connection = [[NSURLConnection alloc]initWithRequest:request delegate:self]; [request release]; [connection release]; I am releasing all my objects as far as I can see but it's still showing this as the source of the 4 memory leaks. This is on the Device running 3.1.3 Is it acceptable to have a few memory leaks in your app or do they all have to go?

  Read the article

• C# performance varying due to memory

  - by user1107474

  Hope this is a valid post here, its a combination of C# issues and hardware. I am benchmarking our server because we have found problems with the performance of our quant library (written in C#). I have simulated the same performance issues with some simple C# code- performing very heavy memory-usage. The code below is in a function which is spawned from a threadpool, up to a maximum of 32 threads (because our server has 4x CPUs x 8 cores each). This is all on .Net 3.5 The problem is that we are getting wildly differing performance. I run the below function 1000 times. The average time taken for the code to run could be, say, 3.5s, but the fastest will only be 1.2s and the slowest will be 7s- for the exact same function! I have graphed the memory usage against the timings and there doesnt appear to be any correlation with the GC kicking in. One thing I did notice is that when running in a single thread the timings are identical and there is no wild deviation. I have also tested CPU-bound algorithms and the timings are identical too. This has made us wonder if the memory bus just cannot cope. I was wondering could this be another .net or C# problem, or is it something related to our hardware? Would this be the same experience if I had used C++, or Java?? We are using 4x Intel x7550 with 32GB ram. Is there any way around this problem in general? Stopwatch watch = new Stopwatch(); watch.Start(); List<byte> list1 = new List<byte>(); List<byte> list2 = new List<byte>(); List<byte> list3 = new List<byte>(); int Size1 = 10000000; int Size2 = 2 * Size1; int Size3 = Size1; for (int i = 0; i < Size1; i++) { list1.Add(57); } for (int i = 0; i < Size2; i = i + 2) { list2.Add(56); } for (int i = 0; i < Size3; i++) { byte temp = list1.ElementAt(i); byte temp2 = list2.ElementAt(i); list3.Add(temp); list2[i] = temp; list1[i] = temp2; } watch.Stop(); (the code is just meant to stress out the memory) I would include the threadpool code, but we used a non-standard threadpool library. EDIT: I have reduced "size1" to 100000, which basically doesn't use much memory and I still get a lot of jitter. This suggests it's not the amount of memory being transferred, but the frequency of memory grabs?

  Read the article

• Ways to avoid Memory Leaks in C/C++

  - by Ankur

  What are some tips I can use to avoid memory leaks in my applications? In my current project I use a tool "INSURE++" which finds the memory leak and generate the report. Apart from the tool is there any method to identify memory leaks and overcome it.

  Read the article

• c# : simulate memory leaks..

  - by dotnet-practitioner

  Hi, I would like to write the following code in c#. a) small console application that simulates memory leak. b) small console application that would invoke the above application and release it right away simulating managing memory leak problem.. In other words the (b) application would continuously call and release application (a) to simulate how the "rebellious" memory leak application is being contained with out addressing the root cause which is application (a). Some sample code for application (a) and (b) would be very helpful. Thanks

  Read the article

• Emgu CV - memory-leaks (memory consumption)

  - by martin pilch

  I am using EmguCV, the OpenCV wrapper for .NET. I am disposing all created objects but my app is still using more and more memory (in release configuration too). I have debugged my app using .NET Memory profiler and get this result: http://img532.imageshack.us/img532/2503/screenqv.png all objects instance count is oscilating but GChandle instance counr is increasing until my machine is unusable. Garbage collector does not release memory (i think). I am using VS 2008 professional, Win7 prof 32-bit, both up to date, and last stable version of emguCV. I can post some app code, if it will help. Thanks and sorry for my English. Martin

  Read the article

• Why does my Delphi program's memory continue to grow?

  - by lkessler

  I am using Delphi 2009 which has the FastMM4 memory manager built into it. My program reads in and processes a large dataset. All memory is freed correctly whenever I clear the dataset or exit the program. It has no memory leaks at all. Using the CurrentMemoryUsage routine given in spenwarr's answer to: http://stackoverflow.com/questions/437683/how-to-get-the-memory-used-by-a-delphi-program, I have displayed the memory used by FastMM4 during processing. What seems to be happening is that memory is use is growing after every process and release cycle. e.g.: 1,456 KB used after starting my program with no dataset. 218,455 KB used after loading a large dataset. 71,994 KB after clearing the dataset completely. If I exit at this point (or any point in my example), no memory leaks are reported. 271,905 KB used after loading the same dataset again. 125,443 KB after clearing the dataset completely. 325,519 KB used after loading the same dataset again. 179,059 KB after clearing the dataset completely. 378,752 KB used after loading the same dataset again. It seems that my program's memory use is growing by about 53,400 KB upon each load/clear cycle. Task Manager confirms that this is actually happening. I have heard that FastMM4 does not always release all of the program's memory back to the Operating system when objects are freed so that it can keep some memory around when it needs more. But this continual growing bothers me. Since no memory leaks are reported, I can't identify a problem. Does anyone know why this is happening, if it is bad, and if there is anything I can or should do about it?

  Read the article

• Windows Physical Direct Memory Mapping

  - by chrisjleaf

  I'm a bit disappointed there is almost no discussion of this no matter where I look so I guess I'll have to ask. I'm writing a cross platform memory bench marking application which requires direct physical address mapping rather than virtual addressing. EDIT The solution would look something like the Linux/Unix system calls: int fd = open("/dev/mem", O_RDONLY); mmap(NULL, len, PROT_READ, MAP_SHARED, fd, PHYSICAL_ADDRESS_OFFSET); which will require the kernel to either give you a virtual page mapping to the desired physical address or return that it failed. This does require supervisor privileges but that is ok. I have seen a lot of information about shared memory and memory mapped files but all of these reside on disc and are thus not really useful when I'm trying to make a system dependent read. It is very similar to writing an IO driver although I do no need write permissions to the physical address. This site gives an example of how to do it on a driver level using the Windows Driver Kit: NT Insider: Sharing Memory between drivers and applications This solution would probably require Visual Studio which currently I do not have access to. (I have downloaded the WDK api but it complained about my use of GCC for Windows). I'm traditionally a Linux programmer so I'm hoping there might be something really simple I'm missing. Thanks in advance if you know something I don't!

  Read the article

• G++ Multi-platform memory leak detection tool

  - by indyK1ng

  Does anyone know where I can find a memory memory leak detection tool for C++ which can be either run in a command line or as an Eclipse plug-in in Windows and Linux. I would like it to be easy to use. Preferably one that doesn't overwrite new(), delete(), malloc() or free(). Something like GDB if its gonna be in the command line, but I don't remember that being used for detecting memory leaks. If there is a unit testing framework which does this automatically, that would be great. This question is similar to other questions (such as http://stackoverflow.com/questions/283726/memory-leak-detection-under-windows-for-gnu-c-c ) however I feel it is different because those ask for windows specific solutions or have solutions which I would rather avoid. I feel I am looking for something a bit more specific here. Suggestions don't have to fulfill all requirements, but as many as possible would be nice. Thanks. EDIT: Since this has come up, by "overwrite" I mean anything which requires me to #include a library or which otherwise changes how C++ compiles my code, if it does this at run time so that running the code in a different environment won't affect anything that would be great. Also, unfortunately, I don't have a Mac, so any suggestions for that are unhelpful, but thank you for trying. My desktop runs Windows (I have Linux installed but my dual monitors don't work with it) and I'd rather not run Linux in a VM, although that is certainly an option. My laptop runs Linux, so I can use that tool on there, although I would definitely prefer sticking to my desktop as the screen space is excellent for keeping all of the design documentation and requirements in view without having to move too much around on the desktop. NOTE: While I may try answers, I won't mark one as accepted until I have tried the suggestion and it is satisfactory. EDIT2: I'm not worried about the cross-platform compatibility of my code, it's a command line application using just the C++ libraries.

  Read the article

• How is an array stored in memory?

  - by George

  In an interest to delve deeper into how memory is allocated and stored, I have written an application that can scan memory address space, find a value, and write out a new value. I developed a sample application with the end goal to be able to programatically locate my array, and overwrite it with a new sequence of numbers. In this situation, I created a single dimensional array, with 5 elements, e.g. int[] array = new int[] {8,7,6,5,4}; I ran my application and searched for a sequence of the five numbers above. I was looking for any value that fell between 4 and 8, for a total of 5 numbers in a row. Unforuntately, my the sequential numbers in my array matched hundreds of results, as the numbers 4 through 8, in no particular sequence happened to be next to each other, in memory, in many situations. Is there any way to distinguish that a set of numbers within memory, represents an array, not simply integers that are next to each other? Is there any way of knowing that if I find a certain value, that the matching values proceeding it are that of an array? I would assume that when I declare int[] array, its pointing at the first address of my array, which would provide some kind of meta-data to what existed in the array, e.g. 0x123456789 meta-data, 5 - 32 bit integers 0x123456789 + 32 "8" 0x123456789 + 64 "7" 0x123456789 + 96 "6" 0x123456789 + 128 "5" 0x123456789 + 160 "4" Am I way off base?

  Read the article

• Firefox consumes too much memory

  - by Vivek

  I have firefox version 11.0 and am running ubuntu 11.10. Firefox takes upto 850MB RAM with only six or seven tabs opened and all the tabs loaded with light weight websites only. I wonder why would a browser consume so much memory. It keeps increasing its memory consumption over time. I have 3GB RAM and most of the times firefox consumes upto 30% of my memory. How do I fix this? EDIT: The output of the command sudo iotop -oPa as asked by @Jippie

  Read the article

• EBS: OPP Out of memory issue...

  - by ashish.shrivastava

  FO Processor is little more hungry for memory compare to other Java process. If XSLT scalable option is not set and the same time your RTF template is not well optimized definitely you are going to hit Out of memory exception while working with large volume of data. If the memory requirement is not too bad, you can set the OOP Heap size using following SQL queries. Check the current OPP JVM Heap size using following SQL query SQL select DEVELOPER_PARAMETERS from FND_CP_SERVICES where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME = 'FNDCPOPP' DEVELOPER_PARAMETERS ----------------------------------------------------- J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m Set the JVM Heap size using following SQL query SQL update FND_CP_SERVICES set DEVELOPER_PARAMETERS = 'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx2048m' where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME = 'FNDCPOPP'); SQLCommit; . You need to restart the Concurrent Manager to make it effective. If this does not resolve the issue, You need to optimize RTF template and set the XSLT scalable option true.

  Read the article

• Getting started with Oracle Database In-Memory Part III - Querying The IM Column Store

  - by Maria Colgan

  In my previous blog posts, I described how to install, enable, and populate the In-Memory column store (IM column store). This weeks post focuses on how data is accessed within the IM column store. Let’s take a simple query “What is the most expensive air-mail order we have received to date?” SELECT Max(lo_ordtotalprice) most_expensive_order FROM lineorderWHERE  lo_shipmode = 5; The LINEORDER table has been populated into the IM column store and since we have no alternative access paths (indexes or views) the execution plan for this query is a full table scan of the LINEORDER table. You will notice that the execution plan has a new set of keywords “IN MEMORY" in the access method description in the Operation column. These keywords indicate that the LINEORDER table has been marked for INMEMORY and we may use the IM column store in this query. What do I mean by “may use”? There are a small number of cases were we won’t use the IM column store even though the object has been marked INMEMORY. This is similar to how the keyword STORAGE is used on Exadata environments. You can confirm that the IM column store was actually used by examining the session level statistics, but more on that later. For now let's focus on how the data is accessed in the IM column store and why it’s faster to access the data in the new column format, for analytical queries, rather than the buffer cache. There are four main reasons why accessing the data in the IM column store is more efficient. 1. Access only the column data needed The IM column store only has to scan two columns – lo_shipmode and lo_ordtotalprice – to execute this query while the traditional row store or buffer cache has to scan all of the columns in each row of the LINEORDER table until it reaches both the lo_shipmode and the lo_ordtotalprice column. 2. Scan and filter data in it's compressed format When data is populated into the IM column it is automatically compressed using a new set of compression algorithms that allow WHERE clause predicates to be applied against the compressed formats. This means the volume of data scanned in the IM column store for our query will be far less than the same query in the buffer cache where it will scan the data in its uncompressed form, which could be 20X larger. 3. Prune out any unnecessary data within each column The fastest read you can execute is the read you don’t do. In the IM column store a further reduction in the amount of data accessed is possible due to the In-Memory Storage Indexes(IM storage indexes) that are automatically created and maintained on each of the columns in the IM column store. IM storage indexes allow data pruning to occur based on the filter predicates supplied in a SQL statement. An IM storage index keeps track of minimum and maximum values for each column in each of the In-Memory Compression Unit (IMCU). In our query the WHERE clause predicate is on the lo_shipmode column. The IM storage index on the lo_shipdate column is examined to determine if our specified column value 5 exist in any IMCU by comparing the value 5 to the minimum and maximum values maintained in the Storage Index. If the value 5 is outside the minimum and maximum range for an IMCU, the scan of that IMCU is avoided. For the IMCUs where the value 5 does fall within the min, max range, an additional level of data pruning is possible via the metadata dictionary created when dictionary-based compression is used on IMCU. The dictionary contains a list of the unique column values within the IMCU. Since we have an equality predicate we can easily determine if 5 is one of the distinct column values or not. The combination of the IM storage index and dictionary based pruning, enables us to only scan the necessary IMCUs. 4. Use SIMD to apply filter predicates For the IMCU that need to be scanned Oracle takes advantage of SIMD vector processing (Single Instruction processing Multiple Data values). Instead of evaluating each entry in the column one at a time, SIMD vector processing allows a set of column values to be evaluated together in a single CPU instruction. The column format used in the IM column store has been specifically designed to maximize the number of column entries that can be loaded into the vector registers on the CPU and evaluated in a single CPU instruction. SIMD vector processing enables the Oracle Database In-Memory to scan billion of rows per second per core versus the millions of rows per second per core scan rate that can be achieved in the buffer cache. I mentioned earlier in this post that in order to confirm the IM column store was used; we need to examine the session level statistics. You can monitor the session level statistics by querying the performance views v$mystat and v$statname. All of the statistics related to the In-Memory Column Store begin with IM. You can see the full list of these statistics by typing: display_name format a30 SELECT display_name FROM v$statname WHERE  display_name LIKE 'IM%'; If we check the session statistics after we execute our query the results would be as follow; SELECT Max(lo_ordtotalprice) most_expensive_order FROM lineorderWHERE lo_shipmode = 5; SELECT display_name FROM v$statname WHERE  display_name IN ('IM scan CUs columns accessed',                        'IM scan segments minmax eligible',                        'IM scan CUs pruned'); As you can see, only 2 IMCUs were accessed during the scan as the majority of the IMCUs (44) in the LINEORDER table were pruned out thanks to the storage index on the lo_shipmode column. In next weeks post I will describe how you can control which queries use the IM column store and which don't. +Maria Colgan

  Read the article

• Design review for application facing memory issues

  - by Mr Moose

  I apologise in advance for the length of this post, but I want to paint an accurate picture of the problems my app is facing and then pose some questions below; I am trying to address some self inflicted design pain that is now leading to my application crashing due to out of memory errors. An abridged description of the problem domain is as follows; The application takes in a “dataset” that consists of numerous text files containing related data An individual text file within the dataset usually contains approx 20 “headers” that contain metadata about the data it contains. It also contains a large tab delimited section containing data that is related to data in one of the other text files contained within the dataset. The number of columns per file is very variable from 2 to 256+ columns. The original application was written to allow users to load a dataset, map certain columns of each of the files which basically indicating key information on the files to show how they are related as well as identify a few expected column names. Once this is done, a validation process takes place to enforce various rules and ensure that all the relationships between the files are valid. Once that is done, the data is imported into a SQL Server database. The database design is an EAV (Entity-Attribute-Value) model used to cater for the variable columns per file. I know EAV has its detractors, but in this case, I feel it was a reasonable choice given the disparate data and variable number of columns submitted in each dataset. The memory problem Given the fact the combined size of all text files was at most about 5 megs, and in an effort to reduce the database transaction time, it was decided to read ALL the data from files into memory and then perform the following; perform all the validation whilst the data was in memory relate it using an object model Start DB transaction and write the key columns row by row, noting the Id of the written row (all tables in the database utilise identity columns), then the Id of the newly written row is applied to all related data Once all related data had been updated with the key information to which it relates, these records are written using SqlBulkCopy. Due to our EAV model, we essentially have; x columns by y rows to write, where x can by 256+ and rows are often into the tens of thousands. Once all the data is written without error (can take several minutes for large datasets), Commit the transaction. The problem now comes from the fact we are now receiving individual files containing over 30 megs of data. In a dataset, we can receive any number of files. We’ve started seen datasets of around 100 megs coming in and I expect it is only going to get bigger from here on in. With files of this size, data can’t even be read into memory without the app falling over, let alone be validated and imported. I anticipate having to modify large chunks of the code to allow validation to occur by parsing files line by line and am not exactly decided on how to handle the import and transactions. Potential improvements I’ve wondered about using GUIDs to relate the data rather than relying on identity fields. This would allow data to be related prior to writing to the database. This would certainly increase the storage required though. Especially in an EAV design. Would you think this is a reasonable thing to try, or do I simply persist with identity fields (natural keys can’t be trusted to be unique across all submitters). Use of staging tables to get data into the database and only performing the transaction to copy data from staging area to actual destination tables. Questions For systems like this that import large quantities of data, how to you go about keeping transactions small. I’ve kept them as small as possible in the current design, but they are still active for several minutes and write hundreds of thousands of records in one transaction. Is there a better solution? The tab delimited data section is read into a DataTable to be viewed in a grid. I don’t need the full functionality of a DataTable, so I suspect it is overkill. Is there anyway to turn off various features of DataTables to make them more lightweight? Are there any other obvious things you would do in this situation to minimise the memory footprint of the application described above? Thanks for your kind attention.

  Read the article

• Applications affected by memory performance

  - by robotron

  I'm writing a paper on the topic of applications affected more by memory performance than processor performance. I've got a lot written regarding the gap between the two, however I can't seem to find anything about the applications that might be affected more by memory performance than by processor speed. I suppose these are applications that make a large amount of memory references, but I have no idea what kind of applications would make such large number of references to make it stand out? Perhaps databases? Can you please give me any pointers on how to proceed, some links to papers? I'm really stuck.

  Read the article

• Wheres my memory going?

  - by Stu2000

  My machine keeps 'freezing' before eventaully logging out with all the programs exiting. This is rather annoying, and I think its because I keep running out of memory. I am not running any custom software, just netbeans, chrome etc. (Stuff I usually run on other ubuntu computers without issue). For some reason my memory usage is through the roof as seen here, but I can't quite figure out why. Here is a screenshot which may be useful with htop and gnome-system monitor open as user and as root. I notice that my console-kit-daemon is taking up about a gig of 'virtual memory'. Is that normal? Any tips/advice will be helpful. In the meantime I have ordered 2 x 4 gig ram sticks to try and just throw hardware at the issue.

  Read the article

< Previous Page | 2 3 4 5 6 7 8 9 10 11 12 13  | Next Page >