Search Results

Search found 13619 results on 545 pages for 'memory mapped'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 162/545 | < Previous Page | 158 159 160 161 162 163 164 165 166 167 168 169  | Next Page >

  • USB Virus, "Program too big to fit in memory"?

    - by ApprenticeHacker
    I got an installer for a piece of software via usb from a friend.Now what was weird was that although my friend and I had the same OS, same brand of laptop and I had a newer version (though I don't think this quite matters), the installer was running properly for him but not for me. It just showed a command line window and exited. I ran it via a batch file in which I included "pause" after running it. Here's the screenshot: Later my friend called me and told me that it was the USB that was the problem. It had some kind of virus on it and it corrupted every executable or folder on it , and it renamed all the sub_files inside folders to some weird jargon. He had tried using another USB and the installer worked fine. Now the friend has (unfortunately) gone back to his city and I can't get the installer again from him. My question is: Is there any way to repair the installer executable and run it? and Do you think the virus has infected my PC? (I have run a system scan with my antivirus and it showed nothing but still I'm worried)

    Read the article

  • Computer suddenly dies; screen displays weird flickering lines, then restarts

    - by Imray
    I've been having this terrible problem for a little while and just managed to get a picture of 'dead screen' for the first time and I am posting it to seek help. Randomly, at irregular intervals (typically once a week), while working on something (it's been different things every time) my computer will just suddenly go dead - the screen turns to exactly the picture below (the lines flicker a little bit), it hangs there for a few seconds and then restarts. Obviously this is extremely frustrating and I want to try to stop it. I've searched numerous postings with similar keywords but nothing exactly the same as mine. Does anyone have any idea what might be the cause of this? I would post all my system settings and installed programs but the list is long and I don't know how much relevance each item would be. If you'd like to know something specific, please comment and I'll let you know whatever you need. SPECS C:\Users\Imray>systeminfo Host Name: Imray OS Name: Microsoft Windows 7 Professional OS Version: 6.1.7600 N/A Build 7600 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free Registered Owner: Imray - Owner Registered Organization: Product ID: 00371-152-9333854-85895 Original Install Date: 06/09/1999, 5:45:21 PM System Boot Time: 22/03/2013, 8:58:18 AM System Manufacturer: Gateway System Model: DX4840 System Type: x64-based PC Processor(s): 1 Processor(s) Installed. [01]: Intel64 Family 6 Model 37 Stepping 2 GenuineIntel ~3201 Mhz BIOS Version: American Megatrends Inc. P01-A3 , 17/05/2010 Windows Directory: C:\Windows System Directory: C:\Windows\system32 Boot Device: \Device\HarddiskVolume2 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC-05:00) Eastern Time (US & Canada) Total Physical Memory: 6,135 MB Available Physical Memory: 3,632 MB Virtual Memory: Max Size: 12,268 MB Virtual Memory: Available: 8,114 MB Virtual Memory: In Use: 4,154 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: \\Imray-OWNER Hotfix(s): 4 Hotfix(s) Installed. [01]: KB971033 [02]: KB958559 [03]: KB977206 [04]: KB981889 Network Card(s): 2 NIC(s) Installed. [01]: 802.11n Wireless PCI Express Card LAN Adapter Connection Name: Wireless Network Connection DHCP Enabled: Yes DHCP Server: 192.168.2.1 IP address(es) [01]: 192.168.2.13 [02]: fe80::1df1:5399:6890:91f6 [02]: Microsoft Virtual WiFi Miniport Adapter Connection Name: Wireless Network Connection 2 DHCP Enabled: Yes DHCP Server: N/A IP address(es) Graphics Card Specs Name ATI Radeon HD 5570 PNP Device ID PCI\VEN_1002&DEV_68D9&SUBSYS_E142174B&REV_00\4&18A4B35E&0&0008 Adapter Type ATI display adapter (0x68D9), ATI Technologies Inc. compatible Adapter Description ATI Radeon HD 5570 Adapter RAM 1.00 GB (1,073,741,824 bytes) Installed Drivers atiu9p64 aticfx64 aticfx64 atiu9pag aticfx32 aticfx32 atiumd64 atidxx64 atidxx64 atiumdag atidxx32 atidxx32 atiumdva atiumd6a atitmm64 Driver Version 8.700.0.0 INF File oem1.inf (ati2mtag_Evergreen section) Color Planes Not Available Color Table Entries 4294967296 Resolution 1920 x 1080 x 59 hertz Bits/Pixel 32 Memory Address 0xD0000000-0xDFFFFFFF Memory Address 0xFBDE0000-0xFBDFFFFF I/O Port 0x0000D000-0x0000DFFF IRQ Channel IRQ 4294967293 I/O Port 0x000003B0-0x000003BB I/O Port 0x000003C0-0x000003DF Memory Address 0xA0000-0xBFFFF Driver c:\windows\system32\drivers\atikmpag.sys (8.14.1.6095, 181.00 KB (185,344 bytes), 06/09/1999 5:59 PM)

    Read the article

  • "The program is too big to fit in memory" problem?!!

    - by user47038
    Please help me!!! This is most amazing error that i have ever had, we write 100 CD, in these CDs we got one setup.exe file that autorun. in some of these CDs when I explore the cd the setup.exe file become the command prompt file and give me above error. and when I reset the system and put it in another pc it`s ok. if I reset the first pc the autorun.exe run. so I change it different computer, randomly get this problem. I am sure about the virus, because I checked on different pcs and with fully update definition so someone tell me what happen?

    Read the article

  • How does data I/O takes place on USB Flash Memory ?

    - by user35704
    I want to know how is data I/O takes place on flash drives which are typically EEPROM's . I thought so as I was writing a C Program that involves file handling . For a normal HDD , that would involve returning the file pointer and reading or writing data to the disk which would be done by read/write HEAD . While in EEPROM's there is no read/write head , as it's works on mnemonic commands , So how come does the C file handling program works when I apply it to a file on flash drive ?

    Read the article

  • VM VirtualBox, Ubuntu 9.10: USB memory stick and SD card not detected...

    - by vikramtheone
    Hello Guys, I have installed Ubuntu9.10 on VM VirtualBox, on my HP laptop. My laptop has several USB ports and also a SD card port. Unfortunately, I'm unable to access both, my pen-drive on the USB port and my 8GB SD card inserted in the SD Card port. Can anyone suggest me what I should do for my Ubuntu to detect both of them? I have installed all latest updates on Ubuntu. I'm able to use my USB mouse. Thank you Vikram

    Read the article

  • Is there a feature in Nagios that allows Memory between checks?

    - by Kyle Brandt
    There are various instances where there are values I want to monitor with Nagios, and I don't care as much about the value itself, but rather how it compares to the previous value. For instance, I wrote one to check the fail counters in OpenVZ. In this case, I didn't care about the value that much, but rather I cared if the value increased. Another example might be switch ports, I would be most interested to get alerted about the change of state of a port (Although perhaps a trap would be better for this one). For my OpenVZ script, I used a temp file, but I am wondering if there is a better way? Maybe Nagios has some variables that plugins (check scripts) can access that are persistent across checks?

    Read the article

  • Is there a USB 3.0 memory stick / thumb drive?

    - by jasondavis
    The new PC I just finished building has USB 3.0 support. I use a USB stick/thumb drive all the time on my PC for stuff, is there an equivalent available anywhere for USB 3.0? Please list one product per post, and provide a link to the manufacturer's product page. Also see the eSATA version of this question.

    Read the article

  • Does uwsgi workers share a common memory ? [ With Nginx ]

    - by Yugal Jindle
    I have configured my Nginx with Django uwsgi. When the django server starts, it reads a 5MB file from the hard-disk. Now, Without Nginx with Django default server python manage.py runserver = Runs immediately and starts serving pages. Problem: With Nginx as the server It takes very long time and several HTTP 504 before it start serving pages. So, How does uwsgi workers work with Nginx ? I have: 4 Workers 512 Threads each So, is the 5MB file getting read 512 * 4 times ? Is there a possible work around for this in Nginx / Uwsgi ?

    Read the article

  • Entity Framework and multi-tenancy database design

    - by Junto
    I am looking at multi-tenancy database schema design for an SaaS concept. It will be ASP.NET MVC - EF, but that isn't so important. Below you can see an example database schema (the Tenant being the Company). The CompanyId is replicated throughout the schema and the primary key has been placed on both the natural key, plus the tenant Id. Plugging this schema into the Entity Framework gives the following errors when I add the tables into the Entity Model file (Model1.edmx): The relationship 'FK_Order_Customer' uses the set of foreign keys '{CustomerId, CompanyId}' that are partially contained in the set of primary keys '{OrderId, CompanyId}' of the table 'Order'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The relationship 'FK_OrderLine_Customer' uses the set of foreign keys '{CustomerId, CompanyId}' that are partially contained in the set of primary keys '{OrderLineId, CompanyId}' of the table 'OrderLine'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The relationship 'FK_OrderLine_Order' uses the set of foreign keys '{OrderId, CompanyId}' that are partially contained in the set of primary keys '{OrderLineId, CompanyId}' of the table 'OrderLine'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The relationship 'FK_Order_Customer' uses the set of foreign keys '{CustomerId, CompanyId}' that are partially contained in the set of primary keys '{OrderId, CompanyId}' of the table 'Order'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The relationship 'FK_OrderLine_Customer' uses the set of foreign keys '{CustomerId, CompanyId}' that are partially contained in the set of primary keys '{OrderLineId, CompanyId}' of the table 'OrderLine'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The relationship 'FK_OrderLine_Order' uses the set of foreign keys '{OrderId, CompanyId}' that are partially contained in the set of primary keys '{OrderLineId, CompanyId}' of the table 'OrderLine'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The relationship 'FK_OrderLine_Product' uses the set of foreign keys '{ProductId, CompanyId}' that are partially contained in the set of primary keys '{OrderLineId, CompanyId}' of the table 'OrderLine'. The set of foreign keys must be fully contained in the set of primary keys, or fully not contained in the set of primary keys to be mapped to a model. The question is in two parts: Is my database design incorrect? Should I refrain from these compound primary keys? I'm questioning my sanity regarding the fundamental schema design (frazzled brain syndrome). Please feel free to suggest the 'idealized' schema. Alternatively, if the database design is correct, then is EF unable to match the keys because it perceives these foreign keys as a potential mis-configured 1:1 relationships (incorrectly)? In which case, is this an EF bug and how can I work around it?

    Read the article

  • Java - Error Message Help

    - by Brian
    In the Code, mem is a of Class Memory and getMDR and getMAR ruturn ints. When I try to compile the code I get the following errors.....how can I fix this? Computer.java:25: write(int,int) in Memory cannot be applied to (int) Input.getInt(mem.write(cpu.getMDR())); ^ Computer.java:28: write(int,int) in Memory cannot be applied to (int) mem.write(cpu.getMAR()); Here is the code for Computer: class Computer{ private Cpu cpu; private Input in; private OutPut out; private Memory mem; public Computer() { Memory mem = new Memory(100); Input in = new Input(); OutPut out = new OutPut(); Cpu cpu = new Cpu(); System.out.println(in.getInt()); } public void run() { cpu.reset(); cpu.setMDR(mem.read(cpu.getMAR())); cpu.fetch2(); while (!cpu.stop()) { cpu.decode(); if (cpu.OutFlag()) OutPut.display(mem.read(cpu.getMAR())); if (cpu.InFlag()) Input.getInt(mem.write(cpu.getMDR())); if (cpu.StoreFlag()) { mem.write(cpu.getMAR()); cpu.getMDR(); } else { cpu.setMDR(mem.read(cpu.getMAR())); cpu.execute(); cpu.fetch(); cpu.setMDR(mem.read(cpu.getMAR())); cpu.fetch2(); } } } Here is the code for Memory: class Memory{ private MemEl[] memArray; private int size; public Memory(int s) {size = s; memArray = new MemEl[s]; for(int i = 0; i < s; i++) memArray[i] = new MemEl(); } public void write (int loc, int val) {if (loc >=0 && loc < size) memArray[loc].write(val); else System.out.println("Index Not in Domain"); } public int read (int loc) {return memArray[loc].read(); } public void dump() { for(int i = 0; i < size; i++) if(i%1 == 0) System.out.println(memArray[i].read()); else System.out.print(memArray[i].read()); } } Here is the code for getMAR and getMDR: public int getMAR() { return ir.getOpcode(); } public int getMDR() { return mdr.read(); }

    Read the article

  • Java map / nio / NFS issue causing a VM fault: "a fault occurred in a recent unsafe memory access op

    - by Matthew Bloch
    I have written a parser class for a particular binary format (nfdump if anyone is interested) which uses java.nio's MappedByteBuffer to read through files of a few GB each. The binary format is just a series of headers and mostly fixed-size binary records, which are fed out to the called by calling nextRecord(), which pushes on the state machine, returning null when it's done. It performs well. It works on a development machine. On my production host, it can run for a few minutes or hours, but always seems to throw "java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code", fingering one of the Map.getInt, getShort methods, i.e. a read operation in the map. The uncontroversial (?) code that sets up the map is this: /** Set up the map from the given filename and position */ protected void open() throws IOException { // Set up buffer, is this all the flexibility we'll need? channel = new FileInputStream(file).getChannel(); MappedByteBuffer map1 = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size()); map1.load(); // we want the whole thing, plus seems to reduce frequency of crashes? map = map1; // assumes the host writing the files is little-endian (x86), ought to be configurable map.order(java.nio.ByteOrder.LITTLE_ENDIAN); map.position(position); } and then I use the various map.get* methods to read shorts, ints, longs and other sequences of bytes, before hitting the end of the file and closing the map. I've never seen the exception thrown on my development host. But the significant point of difference between my production host and development is that on the former, I am reading sequences of these files over NFS (probably 6-8TB eventually, still growing). On my dev machine, I have a smaller selection of these files locally (60GB), but when it blows up on the production host it's usually well before it gets to 60GB of data. Both machines are running java 1.6.0_20-b02, though the production host is running Debian/lenny, the dev host is Ubuntu/karmic. I'm not convinced that will make any difference. Both machines have 16GB RAM, and are running with the same java heap settings. I take the view that if there is a bug in my code, there is enough of a bug in the JVM not to throw me a proper exception! But I think it is just a particular JVM implementation bug due to interactions between NFS and mmap, possibly a recurrence of 6244515 which is officially fixed. I already tried adding in a "load" call to force the MappedByteBuffer to load its contents into RAM - this seemed to delay the error in the one test run I've done, but not prevent it. Or it could be coincidence that was the longest it had gone before crashing! If you've read this far and have done this kind of thing with java.nio before, what would your instinct be? Right now mine is to rewrite it without nio :)

    Read the article

  • javascript simple object creation test: opera leaks?

    - by joe
    Hi, I am trying to figure out certain memory leak conditions in javascript on a few browsers. Currently I'm only testing FF 3.6, Opera 10.10, and Safari 4.0.3. I've started with a fairly simple test, and can confirm no memory leaks in Firefox and Safari. But Opera just takes memory and never gives it back. What gives? Here's the test: <html> <head> <script type="text/javascript"> window.onload = init; //window.onunload = cleanup; var a=[]; function init() { var d = document.createElement('div'); d.innerHTML = "page loading..."; document.body.appendChild(d); for (var i=0; i<400000; i++) { a[i] = new Obj("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"); } d.innerHTML = "PAGE LOADED"; } function cleanup() { for (var i=0; i<400000; i++) { a[i] = null; } } function Obj(msg) { this.msg=msg; } </script> </head> <body> </body> </html> I shouldn't need the cleanup() call on window.unload, but tried that also. No luck. As you can see this is simple JS, no circular DOM links, no closures. I monitor the memory usage using 'top' on Mac 10.4.11. Memory usage spikes up on page load, as expected. In FF and Safari reloading the page does not use any further memory, and all memory is returned when the window (tab) is closed. In Opera, memory spikes on load, and seems to also spike further on each reload (but not always...). But regardless of reload, memory never goes back down below the initial load spike. I had hoped this was a no-brainer test that all browsers would pass, so I could move on to more "interesting" conditions. Am I doing something wrong here? Or is this a known Opera issue? Thanks! -joe

    Read the article

  • deleting element objects of a std vector using erase : a) memory handling and b) better way?

    - by memC
    hi, I have a vec_A that stores instances of class A as: vec_A.push_back(A()); I want to remove some elements in the vector at a later stage and have two questions: a) The element is deleted as: vec_A.erase(iterator) Is there any additional code I need to add to make sure that there is no memory leak? . b) Assume that condition if(num <5) is if num is among a specific numberList. Given this, is there a better way to delete the elements of a vector than what I am illustrating below? #include<vector> #include<stdio.h> #include<iostream> class A { public: int getNumber(); A(int val); ~A(){}; private: int num; }; A::A(int val){ num = val; }; int A::getNumber(){ return num; }; int main(){ int i =0; int num; std::vector<A> vec_A; std::vector<A>::iterator iter; for ( i = 0; i < 10; i++){ vec_A.push_back(A(i)); } iter = vec_A.begin(); while(iter != vec_A.end()){ std::cout << "\n --------------------------"; std::cout << "\n Size before erase =" << vec_A.size(); num = iter->getNumber() ; std::cout << "\n num = "<<num; if (num < 5){ vec_A.erase(iter); } else{ iter++; } std::cout << "\n size after erase =" << vec_A.size(); } std::cout << "\nPress RETURN to continue..."; std::cin.get(); return 0; }

    Read the article

  • Windows debugging - WinDbg

    - by Santhosh77
    Hi, I got the following error while debuggging a process with its core dump. 0:000 !lmi test.exe Loaded Module Info: [test.exe] Module: test Base Address: 00400000 Image Name: test.exe Machine Type: 332 (I386) Time Stamp: 4a3a38ec Thu Jun 18 07:54:04 2009 Size: 27000 CheckSum: 54c30 Characteristics: 10f Debug Data Dirs: Type Size VA Pointer MISC 110, 0, 21000 [Debug data not mapped] FPO 50, 0, 21110 [Debug data not mapped] CODEVIEW 31820, 0, 21160 [Debug data not mapped] - Can't validate symbols, if present. Image Type: FILE - Image read successfully from debugger. test.exe Symbol Type: CV - Symbols loaded successfully from image path. Load Report: cv symbols & lines Does any body know what the error "CODEVIEW 31820, 0, 21160 [Debug data not mapped] - Can't validate symbols, if present." really mean? Is this error meant that i can't read public/private symbols from the executable? If it is not so, why does the WinDbg debugger throws this typr of error? Thanks in advance, Santhosh.

    Read the article

  • Hot to get rid of memory allocations/deallocations in swig wrappers?

    - by Dmitriy Matveev
    I want to use swig for generation of read-only wrappers for a complex object. The object which I want to wrap will always be existent while I will read it. And also I will only use my wrappers at the time that object is existent, thus I don't need any memory management from SWIG. For following swig interface: %module test %immutable; %inline %{ struct Foo { int a; }; struct Bar { int b; Foo f; }; %} I will have a wrappers which will have a lot of garbage in generated interfaces and do useless work which will reduce performance in my case. Generated java wrapper for Bar class will be like this: public class Bar { private long swigCPtr; protected boolean swigCMemOwn; protected Bar(long cPtr, boolean cMemoryOwn) { swigCMemOwn = cMemoryOwn; swigCPtr = cPtr; } protected static long getCPtr(Bar obj) { return (obj == null) ? 0 : obj.swigCPtr; } protected void finalize() { delete(); } public synchronized void delete() { if (swigCPtr != 0) { if (swigCMemOwn) { swigCMemOwn = false; testJNI.delete_Bar(swigCPtr); } swigCPtr = 0; } } public int getB() { return testJNI.Bar_b_get(swigCPtr, this); } public Foo getF() { return new Foo(testJNI.Bar_f_get(swigCPtr, this), true); } public Bar() { this(testJNI.new_Bar(), true); } } I don't need 'swigCMemOwn' field in my wrapper since it always will be false. All code related to this field will also be useless. There are also unnecessary logic in native code: SWIGEXPORT jlong JNICALL Java_some_testJNI_Bar_1f_1get(JNIEnv *jenv, jclass jcls, jlong jarg1, jobject jarg1_) { jlong jresult = 0 ; struct Bar *arg1 = (struct Bar *) 0 ; Foo result; (void)jenv; (void)jcls; (void)jarg1_; arg1 = *(struct Bar **)&jarg1; result = ((arg1)->f); { Foo * resultptr = (Foo *) malloc(sizeof(Foo)); memmove(resultptr, &result, sizeof(Foo)); *(Foo **)&jresult = resultptr; } return jresult; } I don't need these calls to malloc and memmove. I want to force swig to resolve both of these problems, but don't know how. Is it possible?

    Read the article

  • VS 2010 IDE 2GB limt

    - by user561732
    I am using VS 2010 on a win 7 64 bit system with 8 GB of memory. My application is 32 bit. While in the VS 2010 .Net IDE, the app shows up in the Windows task manager as "MyApp.vshost.exe *32" while the VS IDE itself shows up as "devenv.exe *32". I checked and it appears that the VS 2010 IDE file (devenv.exe) is complied with the /LargeAddressAware flag. However, when debugging large models, the IDE fails with an Out of memory exception. In the Windows Task manager, the "MyApp.vshost.exe *32" process indicates about 1400 MB of memory usage (while the "devenv.exe *32" process is well under 500 MB). Is it possible to set the "MyApp.vshost.exe *32" process to be /LargeAddressAware in order to avoid this out of memory situation? If so, how can this be done in the IDE. While setting the final application binary to be /LargeAddressAware would work, I still need to be able to debug the app in the IDE with these type of large models. I should also note that my app has a deep object hierarchy with many collections that together required a lot of memory. However, my issue is not related to trying to create say 1 large array that requires greater then 2 GB of memory etc. I should note that I am able to run the same app in the VB6 IDE and not get an out of memory situation as long as the VB6 IDE is made /LargeAddressAware. In the case of VB6, the IDE and the app being debugged are part of the same process (and not split into 2 as is the case with VS 2010.) The VB6 process can be larger then 3 GB without running into out of memory issues. Ultimately, my objective is to have my app run completely in 64 bit to access more memory. I am hoping that in such cases, the IDE will allow the debugging process to exceed 2 GB without crashing (and certainly more then 1.4 GB as is the current case). However, for now, while 95% of my app is 64 bit, I am calling a legacy COM 32 bit DLL and as such, my entire app is forced to still run in 32 bit mode until I replace that DLL.

    Read the article

  • Bad performance function in PHP. With large files memory blows up! How can I refactor?

    - by André
    Hi I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File. What the function must do is this: Originally I have the CSV like: Copyright (c) 2007 MaxMind LLC. All Rights Reserved. locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,, When I pass the CSV file to this function I got: locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode 1,"O1","","","",0.0000,0.0000,, 2,"AP","","","",35.0000,105.0000,, 3,"EU","","","",47.0000,8.0000,, 4,"AD","","","",42.5000,1.5000,, 5,"AE","","","",24.0000,54.0000,, 6,"AF","","","",33.0000,65.0000,, 7,"AG","","","",17.0500,-61.8000,, 8,"AI","","","",18.2500,-63.1667,, 9,"AL","","","",41.0000,20.0000,, It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory. The function is: public function deleteLine($line_no, $csvFileName) { // this function strips a specific line from a file // if a line is stripped, functions returns True else false // // e.g. // deleteLine(-1, xyz.csv); // strip last line // deleteLine(1, xyz.csv); // strip first line // Assigna o nome do ficheiro $filename = $csvFileName; $strip_return=FALSE; $data=file($filename); $pipe=fopen($filename,'w'); $size=count($data); if($line_no==-1) $skip=$size-1; else $skip=$line_no-1; for($line=0;$line<$size;$line++) if($line!=$skip) fputs($pipe,$data[$line]); else $strip_return=TRUE; return $strip_return; } It is possible to refactor this function to not blow up with the 256MB PHP Memory? Give me some clues. Best Regards,

    Read the article

  • Doing XML extracts with XSLT without having to read the whole DOM tree into memory?

    - by Thorbjørn Ravn Andersen
    I have a situation where I want to extract some information from some very large but regular XML files (just had to do it with a 500 Mb file), and where XSLT would be perfect. Unfortunately those XSLT implementations I am aware of (except the most expensive version of Saxon) does not support only having the necessary part of the DOM read in but reads in the whole tree. This cause the computer to swap to death. The XPath in question is //m/e[contains(.,'foobar') so it is essentially just a grep. Is there an XSLT implementation which can do this? Or an XSLT implementation which given suitable "advice" can do this trick of pruning away the parts in memory which will not be needed again? I'd prefer a Java implementation but both Windows and Linux are viable native platforms. EDIT: The input XML looks like: <log> <!-- Fri Jun 26 12:09:27 CEST 2009 --> <e h='12:09:27,284' l='org.apache.catalina.session.ManagerBase' z='1246010967284' t='ContainerBackgroundProcessor[StandardEngine[Catalina]]' v='10000'> <m>Registering Catalina:type=Manager,path=/axsWHSweb-20090626,host=localhost</m></e> <e h='12:09:27,284' l='org.apache.catalina.session.ManagerBase' z='1246010967284' t='ContainerBackgroundProcessor[StandardEngine[Catalina]]' v='10000'> <m>Force random number initialization starting</m></e> <e h='12:09:27,284' l='org.apache.catalina.session.ManagerBase' z='1246010967284' t='ContainerBackgroundProcessor[StandardEngine[Catalina]]' v='10000'> <m>Getting message digest component for algorithm MD5</m></e> <e h='12:09:27,284' l='org.apache.catalina.session.ManagerBase' z='1246010967284' t='ContainerBackgroundProcessor[StandardEngine[Catalina]]' v='10000'> <m>Completed getting message digest component</m></e> <e h='12:09:27,284' l='org.apache.catalina.session.ManagerBase' z='1246010967284' t='ContainerBackgroundProcessor[StandardEngine[Catalina]]' v='10000'> <m>getDigest() 0</m></e> ...... </log> Essentialy I want to select some m-nodes (and I know the XPath is wrong for that, it was just a quick hack), but maintain the XML layout. EDIT: It appears that STX may be what I am looking for (I can live with another transformation language), and that Joost is an implementation hereof. Any experiences? EDIT: I found that Saxon 6.5.4 with -Xmx1500m could load my XML, so this allowed me to use my XPaths right now. This is just a lucky stroke so I'd still like to solve this generically - this means scriptable which in turn means no handcrafted Java filtering first. EDIT: Oh, by the way. This is a log file very similar to what is generated by the log4j XMLLayout. The reason for XML is to be able to do exactly this, namely do queries on the log. This is the initial try, hence the simple question. Later I'd like to be able to ask more complex questions - therefore I'd like the query language to be able to handle the input file.

    Read the article

  • jQuery memory game: if $('.opened').length; == number run function.

    - by Carl Papworth
    So I'm trying to change the a.heart when there is td.opened == 24. I'm not sure what's going wrong though since nothings happening. HTML: <body> <header> <div id="headerTitle"><a href="index.html">&lt;html<span class="heart">&hearts;</span>ve&gt;</a> </div> <div id="help"> <h2>?</h2> <div id="helpInfo"> <p>How many tiles are there? Let's see [calculating] 25...</p> </div> </div> </header> <div id="reward"> <div id="rewardContainer"> <div id="rewardBG" class="heart">&hearts; </div> <p>OMG, this must be luv<br><a href="index.html" class="exit">x</a></p> </div> </div> <div id="pageWrap"> <div id="mainContent"> <!-- DON'T BE A CHEATER !--> <table id="memory"> <tr> <td class="pair1"><a>&Psi;</a></td> <td class="pair2"><a>&para;</a></td> <td class="pair3"><a>&Xi;</a></td> <td class="pair1"><a>&Psi;</a></td> <td class="pair4"><a >&otimes;</a></td> </tr> <tr> <td class="pair5"><a>&spades;</a></td> <td class="pair6"><a >&Phi;</a></td> <td class="pair7"><a>&sect;</a></td> <td class="pair8"><a>&clubs;</a></td> <td class="pair4"><a>&otimes;</a></td> </tr> <tr> <td class="pair9"><a>&Omega;</a></td> <td class="pair2"><a>&para;</a></td> <td id="goal"> <a href="#reward" class="heart">&hearts;</a> </td> <td class="pair10"><a>&copy;</a></td> <td class="pair9"><a>&Omega;</a></td> </tr> <tr> <td class="pair11"><a>&there4;</a></td> <td class="pair8"><a>&clubs;</a></td> <td class="pair12"><a>&dagger;</a></td> <td class="pair6"><a>&Phi;</a></td> <td class="pair11"><a>&there4;</a></td> </tr> <tr> <td><a class="pair12">&dagger;</a></td> <td><a class="pair5">&spades;</a></td> <td><a class="pair10">&copy;</a></td> <td><a class="pair3">&Xi;</a></td> <td><a class="pair7">&sect;</a></td> </tr> </table> <!-- DON'T BE A CHEATER !--> </div> </div> <!-- END Page Wrap --> <footer> <div class="heartCollection"> <p>collect us if u need luv:<p> <ul> <li><a id="collection1">&hearts;</a></li> <li><a id="collection2">&hearts;</a></li> <li><a id="collection3">&hearts;</a></li> <li><a id="collection4">&hearts;</a></li> <li><a id="collection5">&hearts;</a></li> <li><a id="collection6">&hearts;</a></li> </ul> </div> <div class="credits">with love from Popm0uth ©2012</div> </footer> </body> </html> Javascript: var thisCard = $(this).text(); var activeCard = $('.active').text(); var openedCards = $('.opened').length; $(document).ready(function() { $('a.heart').css('color', '#CCCCCC'); $('a.heart').off('click'); function reset(){ $('td').removeClass('opened'); $('a').removeClass('visible'); $('td').removeClass('active'); }; $('td').click(openCard); function openCard(){ $(this).addClass('opened'); $(this).find('a').addClass('visible'); if ($(".active")[0]){ if ($(this).text() != $('.active').text()) { setTimeout(function(){ reset(); }, 1000); } else { $('.active').removeClass('active'); } } else { $(this).addClass("active"); } if (openedCards == 24){ $(".active").removeClass("active"); $("a.heart").css('color', '#ff63ff'); $("a.heart").off('click'); } } });

    Read the article

  • Advanced Data Source Engine coming to Telerik Reporting Q1 2010

    This is the final blog post from the pre-release series. In it we are going to share with you some of the updates coming to our reporting solution in Q1 2010. A new Declarative Data Source Engine will be added to Telerik Reporting, that will allow full control over data management, and deliver significant gains in rendering performance and memory consumption. Some of the engines new features will be: Data source parameters - those parameters will be used to limit data retrieved from the data source to just the data needed for the report. Data source parameters are processed on the data source side, however only queried data is fetched to the reporting engine, rather than the full data source. This leads to lower memory consumption, because data operations are performed on queried data only, rather than on all data. As a result, only the queried data needs to be stored in the memory vs. the whole dataset, which was the case with the old approach Support for stored procedures - they will assist in achieving a consistent implementation of logic across applications, and are especially practical for performing repetitive tasks. A stored procedure stores the SQL statements and logic, which can then be executed in different reports and/or applications. Stored Procedures will not only save development time, but they will also improve performance, because each stored procedure is compiled on the data base server once, and then is reutilized. In Telerik Reporting, the stored procedure will also be parameterized, where elements of the SQL statement will be bound to parameters. These parameterized SQL queries will be handled through the data source parameters, and are evaluated at run time. Using parameterized SQL queries will improve the performance and decrease the memory footprint of your application, because they will be applied directly on the database server and only the necessary data will be downloaded on the middle tier or client machine; Calculated fields through expressions - with the help of the new reporting engine you will be able to use field values in formulas to come up with a calculated field. A calculated field is a user defined field that is computed "on the fly" and does not exist in the data source, but can perform calculations using the data of the data source object it belongs to. Calculated fields are very handy for adding frequently used formulas to your reports; Improved performance and optimized in-memory OLAP engine - the new data source will come with several improvements in how aggregates are calculated, and memory is managed. As a result, you may experience between 30% (for simpler reports) and 400% (for calculation-intensive reports) in rendering performance, and about 50% decrease in memory consumption. Full design time support through wizards - Declarative data sources are a great advance and will save developers countless hours of coding. In Q1 2010, and true to Telerik Reportings essence, using the new data source engine and its features requires little to no coding, because we have extended most of the wizards to support the new functionality. The newly extended wizards are available in VS2005/VS2008/VS2010 design-time. More features will be revealed on the product's what's new page when the new version is officially released in a few days. Also make sure you attend the free webinar on Thursday, March 11th that will be dedicated to the updates in Telerik Reporting Q1 2010. Did you know that DotNetSlackers also publishes .net articles written by top known .net Authors? We already have over 80 articles in several categories including Silverlight. Take a look: here.

    Read the article

  • Sun Fire X4270 M3 SAP Enhancement Package 4 for SAP ERP 6.0 (Unicode) Two-Tier Standard Sales and Distribution (SD) Benchmark

    - by Brian
    Oracle's Sun Fire X4270 M3 server achieved 8,320 SAP SD Benchmark users running SAP enhancement package 4 for SAP ERP 6.0 with unicode software using Oracle Database 11g and Oracle Solaris 10. The Sun Fire X4270 M3 server using Oracle Database 11g and Oracle Solaris 10 beat both IBM Flex System x240 and IBM System x3650 M4 server running DB2 9.7 and Windows Server 2008 R2 Enterprise Edition. The Sun Fire X4270 M3 server running Oracle Database 11g and Oracle Solaris 10 beat the HP ProLiant BL460c Gen8 server using SQL Server 2008 and Windows Server 2008 R2 Enterprise Edition by 6%. The Sun Fire X4270 M3 server using Oracle Database 11g and Oracle Solaris 10 beat Cisco UCS C240 M3 server running SQL Server 2008 and Windows Server 2008 R2 Datacenter Edition by 9%. The Sun Fire X4270 M3 server running Oracle Database 11g and Oracle Solaris 10 beat the Fujitsu PRIMERGY RX300 S7 server using SQL Server 2008 and Windows Server 2008 R2 Enterprise Edition by 10%. Performance Landscape SAP-SD 2-Tier Performance Table (in decreasing performance order). SAP ERP 6.0 Enhancement Pack 4 (Unicode) Results (benchmark version from January 2009 to April 2012) System OS Database Users SAPERP/ECCRelease SAPS SAPS/Proc Date Sun Fire X4270 M3 2xIntel Xeon E5-2690 @2.90GHz 128 GB Oracle Solaris 10 Oracle Database 11g 8,320 20096.0 EP4(Unicode) 45,570 22,785 10-Apr-12 IBM Flex System x240 2xIntel Xeon E5-2690 @2.90GHz 128 GB Windows Server 2008 R2 EE DB2 9.7 7,960 20096.0 EP4(Unicode) 43,520 21,760 11-Apr-12 HP ProLiant BL460c Gen8 2xIntel Xeon E5-2690 @2.90GHz 128 GB Windows Server 2008 R2 EE SQL Server 2008 7,865 20096.0 EP4(Unicode) 42,920 21,460 29-Mar-12 IBM System x3650 M4 2xIntel Xeon E5-2690 @2.90GHz 128 GB Windows Server 2008 R2 EE DB2 9.7 7,855 20096.0 EP4(Unicode) 42,880 21,440 06-Mar-12 Cisco UCS C240 M3 2xIntel Xeon E5-2690 @2.90GHz 128 GB Windows Server 2008 R2 DE SQL Server 2008 7,635 20096.0 EP4(Unicode) 41,800 20,900 06-Mar-12 Fujitsu PRIMERGY RX300 S7 2xIntel Xeon E5-2690 @2.90GHz 128 GB Windows Server 2008 R2 EE SQL Server 2008 7,570 20096.0 EP4(Unicode) 41,320 20,660 06-Mar-12 Complete benchmark results may be found at the SAP benchmark website http://www.sap.com/benchmark. Configuration and Results Summary Hardware Configuration: Sun Fire X4270 M3 2 x 2.90 GHz Intel Xeon E5-2690 processors 128 GB memory Sun StorageTek 6540 with 4 * 16 * 300GB 15Krpm 4Gb FC-AL Software Configuration: Oracle Solaris 10 Oracle Database 11g SAP enhancement package 4 for SAP ERP 6.0 (Unicode) Certified Results (published by SAP): Number of benchmark users: 8,320 Average dialog response time: 0.95 seconds Throughput: Fully processed order line: 911,330 Dialog steps/hour: 2,734,000 SAPS: 45,570 SAP Certification: 2012014 Benchmark Description The SAP Standard Application SD (Sales and Distribution) Benchmark is a two-tier ERP business test that is indicative of full business workloads of complete order processing and invoice processing, and demonstrates the ability to run both the application and database software on a single system. The SAP Standard Application SD Benchmark represents the critical tasks performed in real-world ERP business environments. SAP is one of the premier world-wide ERP application providers, and maintains a suite of benchmark tests to demonstrate the performance of competitive systems on the various SAP products. See Also SAP Benchmark Website Sun Fire X4270 M3 Server oracle.com OTN Oracle Solaris oracle.com OTN Oracle Database 11g Release 2 Enterprise Edition oracle.com OTN Disclosure Statement Two-tier SAP Sales and Distribution (SD) standard SAP SD benchmark based on SAP enhancement package 4 for SAP ERP 6.0 (Unicode) application benchmark as of 04/11/12: Sun Fire X4270 M3 (2 processors, 16 cores, 32 threads) 8,320 SAP SD Users, 2 x 2.90 GHz Intel Xeon E5-2690, 128 GB memory, Oracle 11g, Solaris 10, Cert# 2012014. IBM Flex System x240 (2 processors, 16 cores, 32 threads) 7,960 SAP SD Users, 2 x 2.90 GHz Intel Xeon E5-2690, 128 GB memory, DB2 9.7, Windows Server 2008 R2 EE, Cert# 2012016. IBM System x3650 M4 (2 processors, 16 cores, 32 threads) 7,855 SAP SD Users, 2 x 2.90 GHz Intel Xeon E5-2690, 128 GB memory, DB2 9.7, Windows Server 2008 R2 EE, Cert# 2012010. Cisco UCS C240 M3 (2 processors, 16 cores, 32 threads) 7,635 SAP SD Users, 2 x 2.90 GHz Intel Xeon E5-2690, 128 GB memory, SQL Server 2008, Windows Server 2008 R2 DE, Cert# 2012011. Fujitsu PRIMERGY RX300 S7 (2 processors, 16 cores, 32 threads) 7,570 SAP SD Users, 2 x 2.90 GHz Intel Xeon E5-2690, 128 GB memory, SQL Server 2008, Windows Server 2008 R2 EE, Cert# 2012008. HP ProLiant DL380p Gen8 (2 processors, 16 cores, 32 threads) 7,865 SAP SD Users, 2 x 2.90 GHz Intel Xeon E5-2690, 128 GB memory, SQL Server 2008, Windows Server 2008 R2 EE, Cert# 2012012. SAP, R/3, reg TM of SAP AG in Germany and other countries. More info www.sap.com/benchmark

    Read the article

  • How to find and fix performance problems in ORM powered applications

    - by FransBouma
    Once in a while we get requests about how to fix performance problems with our framework. As it comes down to following the same steps and looking into the same things every single time, I decided to write a blogpost about it instead, so more people can learn from this and solve performance problems in their O/R mapper powered applications. In some parts it's focused on LLBLGen Pro but it's also usable for other O/R mapping frameworks, as the vast majority of performance problems in O/R mapper powered applications are not specific for a certain O/R mapper framework. Too often, the developer looks at the wrong part of the application, trying to fix what isn't a problem in that part, and getting frustrated that 'things are so slow with <insert your favorite framework X here>'. I'm in the O/R mapper business for a long time now (almost 10 years, full time) and as it's a small world, we O/R mapper developers know almost all tricks to pull off by now: we all know what to do to make task ABC faster and what compromises (because there are almost always compromises) to deal with if we decide to make ABC faster that way. Some O/R mapper frameworks are faster in X, others in Y, but you can be sure the difference is mainly a result of a compromise some developers are willing to deal with and others aren't. That's why the O/R mapper frameworks on the market today are different in many ways, even though they all fetch and save entities from and to a database. I'm not suggesting there's no room for improvement in today's O/R mapper frameworks, there always is, but it's not a matter of 'the slowness of the application is caused by the O/R mapper' anymore. Perhaps query generation can be optimized a bit here, row materialization can be optimized a bit there, but it's mainly coming down to milliseconds. Still worth it if you're a framework developer, but it's not much compared to the time spend inside databases and in user code: if a complete fetch takes 40ms or 50ms (from call to entity object collection), it won't make a difference for your application as that 10ms difference won't be noticed. That's why it's very important to find the real locations of the problems so developers can fix them properly and don't get frustrated because their quest to get a fast, performing application failed. Performance tuning basics and rules Finding and fixing performance problems in any application is a strict procedure with four prescribed steps: isolate, analyze, interpret and fix, in that order. It's key that you don't skip a step nor make assumptions: these steps help you find the reason of a problem which seems to be there, and how to fix it or leave it as-is. Skipping a step, or when you assume things will be bad/slow without doing analysis will lead to the path of premature optimization and won't actually solve your problems, only create new ones. The most important rule of finding and fixing performance problems in software is that you have to understand what 'performance problem' actually means. Most developers will say "when a piece of software / code is slow, you have a performance problem". But is that actually the case? If I write a Linq query which will aggregate, group and sort 5 million rows from several tables to produce a resultset of 10 rows, it might take more than a couple of milliseconds before that resultset is ready to be consumed by other logic. If I solely look at the Linq query, the code consuming the resultset of the 10 rows and then look at the time it takes to complete the whole procedure, it will appear to me to be slow: all that time taken to produce and consume 10 rows? But if you look closer, if you analyze and interpret the situation, you'll see it does a tremendous amount of work, and in that light it might even be extremely fast. With every performance problem you encounter, always do realize that what you're trying to solve is perhaps not a technical problem at all, but a perception problem. The second most important rule you have to understand is based on the old saying "Penny wise, Pound Foolish": the part which takes e.g. 5% of the total time T for a given task isn't worth optimizing if you have another part which takes a much larger part of the total time T for that same given task. Optimizing parts which are relatively insignificant for the total time taken is not going to bring you better results overall, even if you totally optimize that part away. This is the core reason why analysis of the complete set of application parts which participate in a given task is key to being successful in solving performance problems: No analysis -> no problem -> no solution. One warning up front: hunting for performance will always include making compromises. Fast software can be made maintainable, but if you want to squeeze as much performance out of your software, you will inevitably be faced with the dilemma of compromising one or more from the group {readability, maintainability, features} for the extra performance you think you'll gain. It's then up to you to decide whether it's worth it. In almost all cases it's not. The reason for this is simple: the vast majority of performance problems can be solved by implementing the proper algorithms, the ones with proven Big O-characteristics so you know the performance you'll get plus you know the algorithm will work. The time taken by the algorithm implementing code is inevitable: you already implemented the best algorithm. You might find some optimizations on the technical level but in general these are minor. Let's look at the four steps to see how they guide us through the quest to find and fix performance problems. Isolate The first thing you need to do is to isolate the areas in your application which are assumed to be slow. For example, if your application is a web application and a given page is taking several seconds or even minutes to load, it's a good candidate to check out. It's important to start with the isolate step because it allows you to focus on a single code path per area with a clear begin and end and ignore the rest. The rest of the steps are taken per identified problematic area. Keep in mind that isolation focuses on tasks in an application, not code snippets. A task is something that's started in your application by either another task or the user, or another program, and has a beginning and an end. You can see a task as a piece of functionality offered by your application.  Analyze Once you've determined the problem areas, you have to perform analysis on the code paths of each area, to see where the performance problems occur and which areas are not the problem. This is a multi-layered effort: an application which uses an O/R mapper typically consists of multiple parts: there's likely some kind of interface (web, webservice, windows etc.), a part which controls the interface and business logic, the O/R mapper part and the RDBMS, all connected with either a network or inter-process connections provided by the OS or other means. Each of these parts, including the connectivity plumbing, eat up a part of the total time it takes to complete a task, e.g. load a webpage with all orders of a given customer X. To understand which parts participate in the task / area we're investigating and how much they contribute to the total time taken to complete the task, analysis of each participating task is essential. Start with the code you wrote which starts the task, analyze the code and track the path it follows through your application. What does the code do along the way, verify whether it's correct or not. Analyze whether you have implemented the right algorithms in your code for this particular area. Remember we're looking at one area at a time, which means we're ignoring all other code paths, just the code path of the current problematic area, from begin to end and back. Don't dig in and start optimizing at the code level just yet. We're just analyzing. If your analysis reveals big architectural stupidity, it's perhaps a good idea to rethink the architecture at this point. For the rest, we're analyzing which means we collect data about what could be wrong, for each participating part of the complete application. Reviewing the code you wrote is a good tool to get deeper understanding of what is going on for a given task but ultimately it lacks precision and overview what really happens: humans aren't good code interpreters, computers are. We therefore need to utilize tools to get deeper understanding about which parts contribute how much time to the total task, triggered by which other parts and for example how many times are they called. There are two different kind of tools which are necessary: .NET profilers and O/R mapper / RDBMS profilers. .NET profiling .NET profilers (e.g. dotTrace by JetBrains or Ants by Red Gate software) show exactly which pieces of code are called, how many times they're called, and the time it took to run that piece of code, at the method level and sometimes even at the line level. The .NET profilers are essential tools for understanding whether the time taken to complete a given task / area in your application is consumed by .NET code, where exactly in your code, the path to that code, how many times that code was called by other code and thus reveals where hotspots are located: the areas where a solution can be found. Importantly, they also reveal which areas can be left alone: remember our penny wise pound foolish saying: if a profiler reveals that a group of methods are fast, or don't contribute much to the total time taken for a given task, ignore them. Even if the code in them is perhaps complex and looks like a candidate for optimization: you can work all day on that, it won't matter.  As we're focusing on a single area of the application, it's best to start profiling right before you actually activate the task/area. Most .NET profilers support this by starting the application without starting the profiling procedure just yet. You navigate to the particular part which is slow, start profiling in the profiler, in your application you perform the actions which are considered slow, and afterwards you get a snapshot in the profiler. The snapshot contains the data collected by the profiler during the slow action, so most data is produced by code in the area to investigate. This is important, because it allows you to stay focused on a single area. O/R mapper and RDBMS profiling .NET profilers give you a good insight in the .NET side of things, but not in the RDBMS side of the application. As this article is about O/R mapper powered applications, we're also looking at databases, and the software making it possible to consume the database in your application: the O/R mapper. To understand which parts of the O/R mapper and database participate how much to the total time taken for task T, we need different tools. There are two kind of tools focusing on O/R mappers and database performance profiling: O/R mapper profilers and RDBMS profilers. For O/R mapper profilers, you can look at LLBLGen Prof by hibernating rhinos or the Linq to Sql/LLBLGen Pro profiler by Huagati. Hibernating rhinos also have profilers for other O/R mappers like NHibernate (NHProf) and Entity Framework (EFProf) and work the same as LLBLGen Prof. For RDBMS profilers, you have to look whether the RDBMS vendor has a profiler. For example for SQL Server, the profiler is shipped with SQL Server, for Oracle it's build into the RDBMS, however there are also 3rd party tools. Which tool you're using isn't really important, what's important is that you get insight in which queries are executed during the task / area we're currently focused on and how long they took. Here, the O/R mapper profilers have an advantage as they collect the time it took to execute the query from the application's perspective so they also collect the time it took to transport data across the network. This is important because a query which returns a massive resultset or a resultset with large blob/clob/ntext/image fields takes more time to get transported across the network than a small resultset and a database profiler doesn't take this into account most of the time. Another tool to use in this case, which is more low level and not all O/R mappers support it (though LLBLGen Pro and NHibernate as well do) is tracing: most O/R mappers offer some form of tracing or logging system which you can use to collect the SQL generated and executed and often also other activity behind the scenes. While tracing can produce a tremendous amount of data in some cases, it also gives insight in what's going on. Interpret After we've completed the analysis step it's time to look at the data we've collected. We've done code reviews to see whether we've done anything stupid and which parts actually take place and if the proper algorithms have been implemented. We've done .NET profiling to see which parts are choke points and how much time they contribute to the total time taken to complete the task we're investigating. We've performed O/R mapper profiling and RDBMS profiling to see which queries were executed during the task, how many queries were generated and executed and how long they took to complete, including network transportation. All this data reveals two things: which parts are big contributors to the total time taken and which parts are irrelevant. Both aspects are very important. The parts which are irrelevant (i.e. don't contribute significantly to the total time taken) can be ignored from now on, we won't look at them. The parts which contribute a lot to the total time taken are important to look at. We now have to first look at the .NET profiler results, to see whether the time taken is consumed in our own code, in .NET framework code, in the O/R mapper itself or somewhere else. For example if most of the time is consumed by DbCommand.ExecuteReader, the time it took to complete the task is depending on the time the data is fetched from the database. If there was just 1 query executed, according to tracing or O/R mapper profilers / RDBMS profilers, check whether that query is optimal, uses indexes or has to deal with a lot of data. Interpret means that you follow the path from begin to end through the data collected and determine where, along the path, the most time is contributed. It also means that you have to check whether this was expected or is totally unexpected. My previous example of the 10 row resultset of a query which groups millions of rows will likely reveal that a long time is spend inside the database and almost no time is spend in the .NET code, meaning the RDBMS part contributes the most to the total time taken, the rest is compared to that time, irrelevant. Considering the vastness of the source data set, it's expected this will take some time. However, does it need tweaking? Perhaps all possible tweaks are already in place. In the interpret step you then have to decide that further action in this area is necessary or not, based on what the analysis results show: if the analysis results were unexpected and in the area where the most time is contributed to the total time taken is room for improvement, action should be taken. If not, you can only accept the situation and move on. In all cases, document your decision together with the analysis you've done. If you decide that the perceived performance problem is actually expected due to the nature of the task performed, it's essential that in the future when someone else looks at the application and starts asking questions you can answer them properly and new analysis is only necessary if situations changed. Fix After interpreting the analysis results you've concluded that some areas need adjustment. This is the fix step: you're actively correcting the performance problem with proper action targeted at the real cause. In many cases related to O/R mapper powered applications it means you'll use different features of the O/R mapper to achieve the same goal, or apply optimizations at the RDBMS level. It could also mean you apply caching inside your application (compromise memory consumption over performance) to avoid unnecessary re-querying data and re-consuming the results. After applying a change, it's key you re-do the analysis and interpretation steps: compare the results and expectations with what you had before, to see whether your actions had any effect or whether it moved the problem to a different part of the application. Don't fall into the trap to do partly analysis: do the full analysis again: .NET profiling and O/R mapper / RDBMS profiling. It might very well be that the changes you've made make one part faster but another part significantly slower, in such a way that the overall problem hasn't changed at all. Performance tuning is dealing with compromises and making choices: to use one feature over the other, to accept a higher memory footprint, to go away from the strict-OO path and execute queries directly onto the RDBMS, these are choices and compromises which will cross your path if you want to fix performance problems with respect to O/R mappers or data-access and databases in general. In most cases it's not a big issue: alternatives are often good choices too and the compromises aren't that hard to deal with. What is important is that you document why you made a choice, a compromise: which analysis data, which interpretation led you to the choice made. This is key for good maintainability in the years to come. Most common performance problems with O/R mappers Below is an incomplete list of common performance problems related to data-access / O/R mappers / RDBMS code. It will help you with fixing the hotspots you found in the interpretation step. SELECT N+1: (Lazy-loading specific). Lazy loading triggered performance bottlenecks. Consider a list of Orders bound to a grid. You have a Field mapped onto a related field in Order, Customer.CompanyName. Showing this column in the grid will make the grid fetch (indirectly) for each row the Customer row. This means you'll get for the single list not 1 query (for the orders) but 1+(the number of orders shown) queries. To solve this: use eager loading using a prefetch path to fetch the customers with the orders. SELECT N+1 is easy to spot with an O/R mapper profiler or RDBMS profiler: if you see a lot of identical queries executed at once, you have this problem. Prefetch paths using many path nodes or sorting, or limiting. Eager loading problem. Prefetch paths can help with performance, but as 1 query is fetched per node, it can be the number of data fetched in a child node is bigger than you think. Also consider that data in every node is merged on the client within the parent. This is fast, but it also can take some time if you fetch massive amounts of entities. If you keep fetches small, you can use tuning parameters like the ParameterizedPrefetchPathThreshold setting to get more optimal queries. Deep inheritance hierarchies of type Target Per Entity/Type. If you use inheritance of type Target per Entity / Type (each type in the inheritance hierarchy is mapped onto its own table/view), fetches will join subtype- and supertype tables in many cases, which can lead to a lot of performance problems if the hierarchy has many types. With this problem, keep inheritance to a minimum if possible, or switch to a hierarchy of type Target Per Hierarchy, which means all entities in the inheritance hierarchy are mapped onto the same table/view. Of course this has its own set of drawbacks, but it's a compromise you might want to take. Fetching massive amounts of data by fetching large lists of entities. LLBLGen Pro supports paging (and limiting the # of rows returned), which is often key to process through large sets of data. Use paging on the RDBMS if possible (so a query is executed which returns only the rows in the page requested). When using paging in a web application, be sure that you switch server-side paging on on the datasourcecontrol used. In this case, paging on the grid alone is not enough: this can lead to fetching a lot of data which is then loaded into the grid and paged there. Keep note that analyzing queries for paging could lead to the false assumption that paging doesn't occur, e.g. when the query contains a field of type ntext/image/clob/blob and DISTINCT can't be applied while it should have (e.g. due to a join): the datareader will do DISTINCT filtering on the client. this is a little slower but it does perform paging functionality on the data-reader so it won't fetch all rows even if the query suggests it does. Fetch massive amounts of data because blob/clob/ntext/image fields aren't excluded. LLBLGen Pro supports field exclusion for queries. You can exclude fields (also in prefetch paths) per query to avoid fetching all fields of an entity, e.g. when you don't need them for the logic consuming the resultset. Excluding fields can greatly reduce the amount of time spend on data-transport across the network. Use this optimization if you see that there's a big difference between query execution time on the RDBMS and the time reported by the .NET profiler for the ExecuteReader method call. Doing client-side aggregates/scalar calculations by consuming a lot of data. If possible, try to formulate a scalar query or group by query using the projection system or GetScalar functionality of LLBLGen Pro to do data consumption on the RDBMS server. It's far more efficient to process data on the RDBMS server than to first load it all in memory, then traverse the data in-memory to calculate a value. Using .ToList() constructs inside linq queries. It might be you use .ToList() somewhere in a Linq query which makes the query be run partially in-memory. Example: var q = from c in metaData.Customers.ToList() where c.Country=="Norway" select c; This will actually fetch all customers in-memory and do an in-memory filtering, as the linq query is defined on an IEnumerable<T>, and not on the IQueryable<T>. Linq is nice, but it can often be a bit unclear where some parts of a Linq query might run. Fetching all entities to delete into memory first. To delete a set of entities it's rather inefficient to first fetch them all into memory and then delete them one by one. It's more efficient to execute a DELETE FROM ... WHERE query on the database directly to delete the entities in one go. LLBLGen Pro supports this feature, and so do some other O/R mappers. It's not always possible to do this operation in the context of an O/R mapper however: if an O/R mapper relies on a cache, these kind of operations are likely not supported because they make it impossible to track whether an entity is actually removed from the DB and thus can be removed from the cache. Fetching all entities to update with an expression into memory first. Similar to the previous point: it is more efficient to update a set of entities directly with a single UPDATE query using an expression instead of fetching the entities into memory first and then updating the entities in a loop, and afterwards saving them. It might however be a compromise you don't want to take as it is working around the idea of having an object graph in memory which is manipulated and instead makes the code fully aware there's a RDBMS somewhere. Conclusion Performance tuning is almost always about compromises and making choices. It's also about knowing where to look and how the systems in play behave and should behave. The four steps I provided should help you stay focused on the real problem and lead you towards the solution. Knowing how to optimally use the systems participating in your own code (.NET framework, O/R mapper, RDBMS, network/services) is key for success as well as knowing what's going on inside the application you built. I hope you'll find this guide useful in tracking down performance problems and dealing with them in a useful way.  

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 158 159 160 161 162 163 164 165 166 167 168 169  | Next Page >