paul hargreaves - Page 7

Optimising Server-Side Paging - Part I

This article by Paul White presents an optimisation for paging through a wide data set.

Writing more efficient xquery code (avoiding redundant iteration)

Here's a simplified version of a problem I'm working on: I have a bunch of xml data that encodes information about people. Each person is uniquely identified by an 'id' attribute, but they may go by many names. For example, in one document, I might find <person id=1>Paul Mcartney</person> <person id=2>Ringo Starr</person> And in another I might find: <person id=1>Sir Paul McCartney</person> <person id=2>Richard Starkey</person> I want to use xquery to produce a new document that lists every name associated with a given id. i.e.: <person id=1> <name>Paul McCartney</name> <name>Sir Paul McCartney</name> <name>James Paul McCartney</name> </person> <person id=2> ... </person> The way I'm doing this now in xquery is something like this (pseudocode-esque): let $ids := distinct-terms( [all the id attributes on people] ) for $id in $ids return <person id={$id}> { for $unique-name in distinct-values ( for $name in ( [all names] ) where $name/@id=$id return $name ) return <name>{$unique-name}</name> } </person> The problem is that this is really slow. I imagine the bottleneck is the innermost loop, which executes once for every id (of which there are about 1200). I'm dealing with a fair bit of data (300 MB, spread over about 800 xml files), so even a single execution of the query in the inner loop takes about 12 seconds, which means that repeating it 1200 times will take about 4 hours (which might be optimistic - the process has been running for 3 hours so far). Not only is it slow, it's using a whole lot of virtual memory. I'm using Saxon, and I had to set java's maximum heap size to 10 GB (!) to avoid getting out of memory errors, and it's currently using 6 GB of physical memory. So here's how I'd really like to do this (in Pythonic pseudocode): persons = {} for id in ids: person[id] = set() for person in all_the_people_in_my_xml_document: persons[person.id].add(person.name) There, I just did it in linear time, with only one sweep of the xml document. Now, is there some way to do something similar in xquery? Surely if I can imagine it, a reasonable programming language should be able to do it (he said quixotically). The problem, I suppose, is that unlike Python, xquery doesn't (as far as I know) have anything like an associative array. Is there some clever way around this? Failing that, is there something better than xquery that I might use to accomplish my goal? Because really, the computational resources I'm throwing at this relatively simple problem are kind of ridiculous.

Search Results

Search found 1905 results on 77 pages for 'paul hargreaves'.

Page 7/77 | < Previous Page | 3 4 5 6 7 8 9 10 11 12 13 14 | Next Page >

- by Coquelicot

- by Paul Adams

- by Paul

- by Paul Peelen

- by Paul

- by Paul Connolly

- by Paul Johnson

- by Paul Speranza

- by Paul Peelen

- by Paul Nearney

- by Paul

- by Paul Connolly

- by Paul

- by Paul H

- by Paul Hanssen

- by Paul Gordon

- by Paul Johnson

- by Paul Johnson

- by Paul Eisner

- by Paul Maidment

- by Paul Peelen

- by Paul Connolly

- by paul-ogrady

- by Paul

< Previous Page | 3 4 5 6 7 8 9 10 11 12 13 14 | Next Page >