Search Results

Search found 80052 results on 3203 pages for 'data load performance'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 539/3203 | < Previous Page | 535 536 537 538 539 540 541 542 543 544 545 546  | Next Page >

  • Using Hibernate's ScrollableResults to slowly read 90 million records

    - by at
    I simply need to read each row in a table in my MySQL database using Hibernate and write a file based on it. But there are 90 million rows and they are pretty big. So it seemed like the following would be appropriate: ScrollableResults results = session.createQuery("SELECT person FROM Person person") .setReadOnly(true).setCacheable(false).scroll(ScrollMode.FORWARD_ONLY); while (results.next()) storeInFile(results.get()[0]); The problem is the above will try and load all 90 million rows into RAM before moving on to the while loop... and that will kill my memory with OutOfMemoryError: Java heap space exceptions :(. So I guess ScrollableResults isn't what I was looking for? What is the proper way to handle this? I don't mind if this while loop takes days (well I'd love it to not). I guess the only other way to handle this is to use setFirstResult and setMaxResults to iterate through the results and just use regular Hibernate results instead of ScrollableResults. That feels like it will be inefficient though and will start taking a ridiculously long time when I'm calling setFirstResult on the 89 millionth row... UPDATE: setFirstResult/setMaxResults doesn't work, it turns out to take an unusably long time to get to the offsets like I feared. There must be a solution here! Isn't this a pretty standard procedure?? I'm willing to forgo Hibernate and use JDBC or whatever it takes. UPDATE 2: the solution I've come up with which works ok, not great, is basically of the form: select * from person where id > <offset> and <other_conditions> limit 1 Since I have other conditions, even all in an index, it's still not as fast as I'd like it to be... so still open for other suggestions..

    Read the article

  • SQLServer:Namespaces preventing access to query data

    - by Brian
    Hi A beginners question, hopefully easily answered. I've got an xml file I want to load into SQLServer 2008 and extract the useful informaiton. I'm starting simple and just trying to extract the name (\gpx\name). The code I have is: DECLARE @x xml; SELECT @x = xCol.BulkColumn FROM OPENROWSET (BULK 'C:\Data\EM.gpx', SINGLE_BLOB) AS xCol; -- confirm the xml data is in @x select @x as XML_Data -- try and get the name of the gpx section SELECT c.value('name[1]', 'varchar(200)') as Name from @x.nodes('gpx') x(c) Below is a heavily shortened version of the xml file: <?xml version="1.0" encoding="utf-8"?> <gpx xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" version="1.0" creator="Groundspeak Pocket Query" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.groundspeak.com/cache/1/0 http://www.groundspeak.com/cache/1/0/cache.xsd" xmlns="http://www.topografix.com/GPX/1/0"> <name>EM</name> <desc>Geocache file generated by Groundspeak</desc> <author>Groundspeak</author> <email>[email protected]</email> <time>2010-03-24T14:01:36.4931342Z</time> <keywords>cache, geocache, groundspeak</keywords> <wpt lat="51.2586" lon="-2.213067"> <time>2008-03-30T07:00:00Z</time> <name>GC1APHM</name> <desc>Sandman's Noble Hoard by Sandman1973, Unknown Cache (2/3)</desc> <groundspeak:cache id="832000" available="True" archived="False" xmlns:groundspeak="http://www.groundspeak.com/cache/1/0"> <groundspeak:name>Sandman's Noble Hoard</groundspeak:name> <groundspeak:placed_by>Sandman1973</groundspeak:placed_by> </groundspeak:cache> </wpt> </gpx> If the first two lines are replaced with just: <gpx> the above example works correctly, however I then can't access groundspeak:name (/gpx/wpt/groundspeak:cache/groundspeak:name), so my guess its a problem with the namespace. Any help would be appriciated.

    Read the article

  • Retrive unchanged data

    - by cnkt
    How can i get the Doctrine_Record's unchanged version of field data. For example; echo $user->username; // Prints 'David' $user->username = 'John'; echo $user->username; // Prints 'John' How to get the pre-changed value (David)?

    Read the article

  • Django - testing using large tables of static data

    - by Michael B
    I am using "manage.py test" along with a JSON fixture I created using using 'dumpdata' My problem is that several of the tables in the fixture are very large (for example one containing the names of all cities in the US) which makes running a test incredibly slow. Seeing as several of these tables are never modified by the program (eg - the city names will never need to be modified), it doesn't make much sense to create and tear down these tables for every test run. Is there a better way to be testing this code using this kind of data?

    Read the article

  • Large data processing in x86 C# gives System.OutOfMemory exception

    - by Cool
    I am processing XML coming from server which contains both images and data in one C# function (compiled 32 bit). When I try to parse this xml in memory it gives me System.OutOfMemory exception. Is there any way to avoid this error? My guess is, system cannot find contiguous block of 50-100MB memory. (my pc hv 8Gig ram and its quad core)

    Read the article

  • how to search data between specified dates

    - by sumit
    i have a query OleDbCommand com = new OleDbCommand("select * from techs where actd0v between '" + TextBox1.Text + "' and '" + TextBox2.Text + "'" , con); where textbox 1 and 2 are the specified dates in which i want to retreive some date related to those dates.but when i m trying to find the data between the given dates it gives me some dates which are not included in between them.is there any pattern i need to specify to get all the dates which occur in between the specified dates. i m using ms access database for my project.

    Read the article

  • Cluster Graph Visualization using python

    - by AlgoMan
    I am assembling different visualization tools that are available in python language. I found the Treemap. (http://pypi.python.org/pypi/treemap/1.05) Can you suggest some other tools that are available. I am exploring different ways of visualization of web data.

    Read the article

  • Best practices for model driven development using LiveCycle Data Services

    - by Adnan
    What are your advises on using model driven development in developing enterprise applications. Adobe's LiveCycle Data Services looks very promising, I have found numerous tutorials/videos that shows how fast an application can be build by having methods/functions auto-generated. What are the best-practices, is it good/bad to use those auto-generated methods, they can really save a lot of time. All suggestions are welcome, also if you know some existing blog/discussion please let me know.

    Read the article

  • Passing a ManagedObjectContext to a second view

    - by amo
    I'm writing my first iPhone/Cocoa app. It has two table views inside a navigation view. When you touch a row in the first table view, you are taken to the second table view. I would like the second view to display records from the CoreData entities related to the row you touched in the first view. I have the CoreData data showing up fine in the first table view. You can touch a row and go to the second table view. I'm able to pass info from the selected object from the first to the second view. But I cannot get the second view to do its own CoreData fetching. For the life of me I cannot get the managedObjectContext object to pass to the second view controller. I don't want to do the lookups in the first view and pass a dictionary because I want to be able to use a search field to refine results in the second view, as well as insert new entries to the CoreData data from there. Here's the function that transitions from the first to the second view. - (void)tableView:(UITableView *)tableView didSelectRowAtIndexPath:(NSIndexPath *)indexPath { // Navigation logic may go here -- for example, create and push another view controller. NSManagedObject *selectedObject = [[self fetchedResultsController] objectAtIndexPath:indexPath]; SecondViewController *secondViewController = [[SecondViewController alloc] initWithNibName:@"SecondView" bundle:nil]; secondViewController.tName = [[selectedObject valueForKey:@"name"] description]; secondViewController.managedObjectContext = [self managedObjectContext]; [self.navigationController pushViewController:secondViewController animated:YES]; [secondViewController release]; } And this is the function inside SecondViewController that crashes: - (void)viewDidLoad { [super viewDidLoad]; self.title = tName; NSError *error; if (![[self fetchedResultsController] performFetch:&error]) { // <-- crashes here // Handle the error... } } - (NSFetchedResultsController *)fetchedResultsController { if (fetchedResultsController != nil) { return fetchedResultsController; } /* Set up the fetched results controller. */ // Create the fetch request for the entity. NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init]; // Edit the entity name as appropriate. // **** crashes on the next line because managedObjectContext == 0x0 NSEntityDescription *entity = [NSEntityDescription entityForName:@"SecondEntity" inManagedObjectContext:managedObjectContext]; [fetchRequest setEntity:entity]; // <snip> ... more code here from Apple template, never gets executed because of the crashing return fetchedResultsController; } Any ideas on what I am doing wrong here? managedObjectContext is a retained property. UPDATE: I inserted a NSLog([[managedObjectContext registeredObjects] description]); in viewDidLoad and it appears managedObjectContext is being passed just fine. Still crashing, though. Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '+entityForName: could not locate an NSManagedObjectModel for entity name 'SecondEntity''

    Read the article

  • Connecting a DomainContext and loading data manually in Silverlight 4 RC application

    - by Redburn
    I am new to WCF & RIA services and have what seems like a basic problem. I have multiple times in my silverlight 4 RC application dragged and dropped from my datasource window to my form and have returned information from the database. However I have a need to query the database for other information (Building a report) When I try to use the following code, I recieve no error, but I also do not get any information back from the service. //Global public UMSDomainContext _umsDomainContext = new UMSDomainContext(); //In the Initialization portion _umsDomainContext.Load(_umsDomainContext.GetUMOptionsQuery()); //Queries var name = from n in _umsDomainContext.UMOptions select n.DistrictName; var street1 = from c in _umsDomainContext.UMOptions select c.Address1; var street2 = from c in _umsDomainContext.UMOptions select c.Address2; var city = from c in _umsDomainContext.UMOptions select c.City; var zip = from c in _umsDomainContext.UMOptions select c.Zip; Im calling the current additional assem. using System.Linq; using System.ServiceModel.DomainServices.Client; using System.ComponentModel.DataAnnotations; using MTT.Web;

    Read the article

  • Delphi - MySQL Best Data aware components to use

    - by Brad
    I need my applicaiton to connect to my web server's MySQL database, what is the best option for this. Perfered Data aware Component. I tried zeos 7, but I keep getting the error: SQL error: Client does not support authentication protocal requested by server; consider upgrading MySQL client and have not been able to fix it. Thanks -Brad

    Read the article

  • How to use jquery to paginate json data?

    - by Pandiya Chendur
    My json Data looks like this {"Table" : [{"Emp_Id" : "3","Identity_No" : "","Emp_Name" : "Jerome","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Supervisior","Desig_Description" : "Supervisior of the Construction","SalaryBasis" : "Monthly","FixedSalary" : "25000.00"},{"Emp_Id" : "4","Identity_No" : "","Emp_Name" : "Mohan","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Acc ","Desig_Description" : "Accountant","SalaryBasis" : "Monthly","FixedSalary" : "200.00"},{"Emp_Id" : "5","Identity_No" : "","Emp_Name" : "Murugan","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Mason","Desig_Description" : "Mason","SalaryBasis" : "Weekly","FixedSalary" : "150.00"},{"Emp_Id" : "6","Identity_No" : "","Emp_Name" : "Ram","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Mason","Desig_Description" : "Mason","SalaryBasis" : "Weekly","FixedSalary" : "120.00"},{"Emp_Id" : "7","Identity_No" : "","Emp_Name" : "Raja","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Mason","Desig_Description" : "Mason","SalaryBasis" : "Weekly","FixedSalary" : "135.00"},{"Emp_Id" : "8","Identity_No" : "","Emp_Name" : "Raja kumar","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Mason Helper","Desig_Description" : "Mason Helper","SalaryBasis" : "Weekly","FixedSalary" : "105.00"},{"Emp_Id" : "9","Identity_No" : "","Emp_Name" : "Lakshmi","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Mason Helper","Desig_Description" : "Mason Helper","SalaryBasis" : "Weekly","FixedSalary" : "100.00"},{"Emp_Id" : "10","Identity_No" : "","Emp_Name" : "Palani","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Carpenter","Desig_Description" : "Carpenter","SalaryBasis" : "Weekly","FixedSalary" : "200.00"},{"Emp_Id" : "11","Identity_No" : "","Emp_Name" : "Annamalai","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Carpenter","Desig_Description" : "Carpenter","SalaryBasis" : "Weekly","FixedSalary" : "220.00"},{"Emp_Id" : "12","Identity_No" : "","Emp_Name" : "David","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Steel Fixer","Desig_Description" : "Steel Fixer","SalaryBasis" : "Weekly","FixedSalary" : "220.00"},{"Emp_Id" : "13","Identity_No" : "","Emp_Name" : "Chandru","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Steel Fixer","Desig_Description" : "Steel Fixer","SalaryBasis" : "Weekly","FixedSalary" : "220.00"},{"Emp_Id" : "14","Identity_No" : "","Emp_Name" : "Mani","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Steel Helper","Desig_Description" : "Steel Helper","SalaryBasis" : "Weekly","FixedSalary" : "175.00"},{"Emp_Id" : "15","Identity_No" : "","Emp_Name" : "Karthik","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Wood Fixer","Desig_Description" : "Wood Fixer","SalaryBasis" : "Weekly","FixedSalary" : "195.00"},{"Emp_Id" : "16","Identity_No" : "","Emp_Name" : "Bala","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Wood Fixer","Desig_Description" : "Wood Fixer","SalaryBasis" : "Weekly","FixedSalary" : "185.00"},{"Emp_Id" : "17","Identity_No" : "","Emp_Name" : "Tamil arasi","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Wood Helper","Desig_Description" : "Wood Helper","SalaryBasis" : "Weekly","FixedSalary" : "185.00"},{"Emp_Id" : "18","Identity_No" : "","Emp_Name" : "Perumal","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Cook","Desig_Description" : "Cook","SalaryBasis" : "Weekly","FixedSalary" : "105.00"},{"Emp_Id" : "19","Identity_No" : "","Emp_Name" : "Andiappan","Address" : "Madurai","Date_Of_Birth" : "","Desig_Name" : "Watchman","Desig_Description" : "Watchman","SalaryBasis" : "Weekly","FixedSalary" : "150.00"}]} There are 22 records in this json... How to paginate this json data 5 per page using jquery? EDIT: The above image is my summary view of employee list iterated using jquery var jsonObj = JSON.parse(HfJsonValue); for (var i = jsonObj.Table.length - 1; i >= 0; i--) { var employee = jsonObj.Table[i]; $('<div class="resultsdiv"><br /><span class="resultName">' + employee.Emp_Name + '</span><span class="resultfields" style="padding-left:100px;">Category&nbsp;:</span>&nbsp;<span class="resultfieldvalues">' + employee.Desig_Name + '</span><br /><br /><span id="SalaryBasis" class="resultfields">Salary Basis&nbsp;:</span>&nbsp;<span class="resultfieldvalues">' + employee.SalaryBasis + '</span><span class="resultfields" style="padding-left:25px;">Salary&nbsp;:</span>&nbsp;<span class="resultfieldvalues">' + employee.FixedSalary + '</span><span style="font-size:110%;font-weight:bolder;padding-left:25px;">Address&nbsp;:</span>&nbsp;<span class="resultfieldvalues">' + employee.Address + '</span></div>').insertAfter('#ResultsDiv'); } I get 22 records now it may grow how to paginate json date by using jquery pagination.. Any suggestion...

    Read the article

  • How does Blippy get its data

    - by Ali
    I was wondering how Blippy is able to get my data? It requires me to put in my bank name, bank card number and password, so is it doing a simple website scrape by logging in? My bank, however also requires a seperate passphrase as well. How does it get around that? Can urllib and such libraries be used in Python to replicate Blippy functionality? site:blippy.com

    Read the article

  • Receiving data with Winsock

    - by Tamir
    Right now, I'm programming the networking for my online game, and I'm not really sure what to do about receiving data. The problem is that I can't really guess the packet's size, so I thought of reading just 4 bytes from the packet and converting them to an int to know what's the packet's size. Then I'll just create a buffer in that size and receive the rest of the packet, is that a good idea? For your information, I'm using non-blocking i/o.

    Read the article

  • How Can I Use This jSON data like array

    - by mTuran
    hi, i have json data like this: {"Adobe Photoshop":"70","Adobe Premier":"71","Animasyon":"72","AutoCAD":"73","Bro\u015f\u00fcr & Bilet":"75","CorelDraw":"76","Flash":"77","Grafik Tasar\u0131m":"68","Karikat\u00fcr":"78","Kartvizit Tasar\u0131m":"79","Logo Tasar\u0131m":"80","Poster":"81","Reklam Tasar\u0131m":"74","Ses D\u00fczenleme":"82","Video D\u00fczenleme":"69","Web Tasar\u0131m":"83"} First think(Adobe Photoshop) means tag name, second thing(70) means tag id. I want to use this information in jquery like array to build checkboxes for every tag. How can i done this ? Thanks

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 535 536 537 538 539 540 541 542 543 544 545 546  | Next Page >