Search Results

Search found 8875 results on 355 pages for 'optimized solutions'.

Page 163/355 | < Previous Page | 159 160 161 162 163 164 165 166 167 168 169 170  | Next Page >

• Would this method work to scale out SQL queries?

  - by David

  I have a database containing a single huge table. At the moment a query can take anything from 10 to 20 minutes and I need that to go down to 10 seconds. I have spent months trying different products like GridSQL. GridSQL works fine, but is using its own parser which does not have all the needed features. I have also optimized my database in various ways without getting the speedup I need. I have a theory on how one could scale out queries, meaning that I utilize several nodes to run a single query in parallel. The idea is to take an incoming SQL query and simply run it exactly like it is on all the nodes. When the results are returned to a coordinator node, the same query is run on the union of the resultsets. I realize that an aggregate function like average need to be rewritten into a count and sum to the nodes and that the coordinator divides the sum of the sums with the sum of the counts to get the average. What kinds of problems could not easily be solved using this model. I believe one issue would be the count distinct function. Edit: I am getting so many nice suggestions, but none have addressed the method.

  Read the article

• Optimization in Python - do's, don'ts and rules of thumb.

  - by JV

  Well I was reading this post and then I came across a code which was: jokes=range(1000000) domain=[(0,(len(jokes)*2)-i-1) for i in range(0,len(jokes)*2)] I thought wouldn't it be better to calculate the value of len(jokes) once outside the list comprehension? Well I tried it and timed three codes jv@Pioneer:~$ python -m timeit -s 'jokes=range(1000000);domain=[(0,(len(jokes)*2)-i-1) for i in range(0,len(jokes)*2)]' 10000000 loops, best of 3: 0.0352 usec per loop jv@Pioneer:~$ python -m timeit -s 'jokes=range(1000000);l=len(jokes);domain=[(0,(l*2)-i-1) for i in range(0,l*2)]' 10000000 loops, best of 3: 0.0343 usec per loop jv@Pioneer:~$ python -m timeit -s 'jokes=range(1000000);l=len(jokes)*2;domain=[(0,l-i-1) for i in range(0,l)]' 10000000 loops, best of 3: 0.0333 usec per loop Observing the marginal difference 2.55% between the first and the second made me think - is the first list comprehension domain=[(0,(len(jokes)*2)-i-1) for i in range(0,len(jokes)*2)] optimized internally by python? or is 2.55% a big enough optimization (given that the len(jokes)=1000000)? If this is - What are the other implicit/internal optimizations in Python ? What are the developer's rules of thumb for optimization in Python? Edit1: Since most of the answers are "don't optimize, do it later if its slow" and I got some tips and links from Triptych and Ali A for the do's. I will change the question a bit and request for don'ts. Can we have some experiences from people who faced the 'slowness', what was the problem and how it was corrected? Edit2: For those who haven't here is an interesting read Edit3: Incorrect usage of timeit in question please see dF's answer for correct usage and hence timings for the three codes.

  Read the article

• Optimizing a shared buffer in a producer/consumer multithreaded environment

  - by Etan

  I have some project where I have a single producer thread which writes events into a buffer, and an additional single consumer thread which takes events from the buffer. My goal is to optimize this thing for a single machine to achieve maximum throughput. Currently, I am using some simple lock-free ring buffer (lock-free is possible since I have only one consumer and one producer thread and therefore the pointers are only updated by a single thread). #define BUF_SIZE 32768 struct buf_t { volatile int writepos; volatile void * buffer[BUF_SIZE]; volatile int readpos;) }; void produce (buf_t *b, void * e) { int next = (b->writepos+1) % BUF_SIZE; while (b->readpos == next); // queue is full. wait b->buffer[b->writepos] = e; b->writepos = next; } void * consume (buf_t *b) { while (b->readpos == b->writepos); // nothing to consume. wait int next = (b->readpos+1) % BUF_SIZE; void * res = b->buffer[b->readpos]; b->readpos = next; return res; } buf_t *alloc () { buf_t *b = (buf_t *)malloc(sizeof(buf_t)); b->writepos = 0; b->readpos = 0; return b; } However, this implementation is not yet fast enough and should be optimized further. I've tried with different BUF_SIZE values and got some speed-up. Additionaly, I've moved writepos before the buffer and readpos after the buffer to ensure that both variables are on different cache lines which resulted also in some speed. What I need is a speedup of about 400 %. Do you have any ideas how I could achieve this using things like padding etc?

  Read the article

• Sybase stored procedure - how do I create an index on a #table?

  - by DVK

  I have a stored procedure which creates and works with a temporary #table Some of the queries would be tremendously optimized if that temporary #table would have an index created on it. However, creating an index within the stored procedure fails: create procedure test1 as SELECT f1, f2, f3 INTO #table1 FROM main_table WHERE 1 = 2 -- insert rows into #table1 create index my_idx on #table1 (f1) SELECT f1, f2, f3 FROM #table1 (index my_idx) WHERE f1 = 11 -- "QUERY X" When I call the above, the query plan for "QUERY X" shows a table scan. If I simply run the code above outside the stored procedure, the messages show the following warning: Index 'my_idx' specified as optimizer hint in the FROM clause of table '#table1' does not exist. Optimizer will choose another index instead. This can be resolved when running ad-hoc (outside the stored procedure) by splitting the code above in two batches by addding "go" after index creation: create index my_idx on #table1 (f1) go Now, "QUERY X" query plan shows the use of index "my_idx". QUESTION: How do I mimique running the "create index" in a separate batch when it's inside the stored procedure? I can't insert a "go" there like I do with the ad-hoc copy above. P.S. If it matters, this is on Sybase 12.

  Read the article

• Query with many CASE statements - optimization

  - by Nemanja Vujacic

  Hi guys, I have one very dirty query that per sure can be optimized because there are so many CASE statements in it! SELECT (CASE pa.KplusTable_Id WHEN 1 THEN sp.sp_id WHEN 2 THEN fw.fw_id WHEN 3 THEN s.sw_Id WHEN 4 THEN id.ia_id END) as Deal_Id, max(CASE pa.KplusTable_Id WHEN 1 THEN sp.Trans_Id WHEN 2 THEN fw.Trans_Id WHEN 3 THEN s.Trans_Id WHEN 4 THEN id.Trans_Id END) as TransId_CurrentMax INTO #MaxRazlicitOdNull FROM #PotencijalniAktuelni pa LEFT JOIN kplus_sp sp (nolock) on sp.sp_id=pa.Deal_Id AND pa.KplusTable_Id=1 LEFT JOIN kplus_fw fw (nolock) on fw.fw_id=pa.Deal_Id AND pa.KplusTable_Id=2 LEFT JOIN dev_sw s (nolock) on s.sw_Id=pa.Deal_Id AND pa.KplusTable_Id=3 LEFT JOIN kplus_ia id (nolock) on id.ia_id=pa.Deal_Id AND pa.KplusTable_Id=4 WHERE isnull(CASE pa.KplusTable_Id WHEN 1 THEN sp.BROJ_TIKETA WHEN 2 THEN fw.BROJ_TIKETA WHEN 3 THEN s.tiket WHEN 4 THEN id.BROJ_TIKETA END, '')<>'' GROUP BY CASE pa.KplusTable_Id WHEN 1 THEN sp.sp_id WHEN 2 THEN fw.fw_id WHEN 3 THEN s.sw_Id WHEN 4 THEN id.ia_id END Because I have same condition couple times, do you have idea how to optimize query, make it simpler and better. All suggestions are welcome! TnX in advance! Nemanja

  Read the article

• How to transform vertical table into horizontal table?

  - by avivo

  Hello, I have one table Person: Id Name 1 Person1 2 Person2 3 Person3 And I have its child table Profile: Id PersonId FieldName Value 1 1 Firstname Alex 2 1 Lastname Balmer 3 1 Email [email protected] 4 1 Phone +1 2 30004000 And I want to get data from these two tables in one row like this: Id Name Firstname Lastname Email Phone 1 Person1 Alex Balmer [email protected] +1 2 30004000 What is the most optimized query to get these vertical (key, value) values in one row like this? Now I have a problem that I done four joins of child table to parent table because I need to get these four fields. Some optimization is for sure possible. I would like to be able to modify this query in easy way when I add new field (key,value). What is the best way to do this? To create some StoreProcedure? I would like to have strongly types in my DB layer (C#) and using LINQ (when programming) so it means when I add some new Key, Value pair in Profile table I would like to do minimal modifications in DB and C# if possible. Actually I am trying to get some best practices in this case.

  Read the article

• String Length Evaluating Incorrectly

  - by Justin R.

  My coworker and I are debugging an issue in a WCF service he's working on where a string's length isn't being evaluated correctly. He is running this method to unit test a method in his WCF service: // Unit test method public void RemoveAppGroupTest() { string addGroup = "TestGroup"; string status = string.Empty; string message = string.Empty; appActiveDirectoryServicesClient.RemoveAppGroup("AOD", addGroup, ref status, ref message); } // Inside the WCF service [OperationBehavior(Impersonation = ImpersonationOption.Required)] public void RemoveAppGroup(string AppName, string GroupName, ref string Status, ref string Message) { string accessOnDemandDomain = "MyDomain"; RemoveAppGroupFromDomain(AppName, accessOnDemandDomain, GroupName, ref Status, ref Message); } public AppActiveDirectoryDomain(string AppName, string DomainName) { if (string.IsNullOrEmpty(AppName)) { throw new ArgumentNullException("AppName", "You must specify an application name"); } } We tried to step into the .NET source code to see what value string.IsNullOrEmpty was receiving, but the IDE printed this message when we attempted to evaluate the variable: 'Cannot obtain value of local or argument 'value' as it is not available at this instruction pointer, possibly because it has been optimized away.' (None of the projects involved have optimizations enabled). So, we decided to try explicitly setting the value of the variable inside the method itself, immediately before the length check -- but that didn't help. // Lets try this again. public AppActiveDirectoryDomain(string AppName, string DomainName) { // Explicitly set the value for testing purposes. AppName = "AOD"; if (AppName == null) { throw new ArgumentNullException("AppName", "You must specify an application name"); } if (AppName.Length == 0) { // This exception gets thrown, even though it obviously isn't a zero length string. throw new ArgumentNullException("AppName", "You must specify an application name"); } } We're really pulling our hair out on this one. Has anyone else experienced behavior like this? Any tips on debugging it?

  Read the article

• Linux time sample based profiler.

  - by Caspin

  short version: Is there a good time based sampling profiler for Linux? long version: I generally use OProfile to optimize my applications. I recently found a shortcoming that has me wondering. The problem was a tight loop spawning c++filt to demangle a c++ name. I only stumbled upon the code by accident while chasing down another bottleneck. The OProfile didn't show anything unusual about the code so I almost ignored it but my code sense told me to optimize the call and see what happened. I changed the popen of c++filt to abi::__cxa_demangle. The runtime went from more than a minute to a little over a second. About a x60 speed up. Is there a way I could have configured OProfile to flag the popen call? As the profile data sits now OProfile thinks the bottle neck was the heap and std::string calls (which BTW once optimized dropped the runtime to less than a second, more than x2 speed up). Here is my OProfile configuration: $ sudo opcontrol --status Daemon not running Event 0: CPU_CLK_UNHALTED:90000:0:1:1 Separate options: library vmlinux file: none Image filter: /path/to/excutable Call-graph depth: 7 Buffer size: 65536 Is there another profiler for Linux that could have found the bottleneck? I suspect the issue is that OProfile only logs its samples to the currently running process. I'd like it to always log its samples to the process I'm profiling. So if the process is currently switched out (blocking on IO or a popen call) OProfile would just place its sample at the blocked call. If I can't fix this, OProfile will only be useful when the executable is pushing near 100% CPU. It can't help with executables that that have inefficient blocking calls.

  Read the article

• C# async callback on disposed form

  - by Rodney Burton

  Quick question: One of my forms in my winform app (c#) makes an async call to a WCF service to get some data. If the form happens to close before the callback happens, it crashes with an error about accessing a disposed object. What's the correct way to check/handle this situation? The error happens on the Invoke call to the method to update my form, but I can't drill down to the inner exception because it says the code has been optimized. The Code: public void RequestUserPhoto(int userID) { WCF.Service.BeginGetUserPhoto(userID, new AsyncCallback(GetUserPhotoCB), userID); } public void GetUserPhotoCB(IAsyncResult result) { var photo = WCF.Service.EndGetUserPhoto(result); int userID = (int)result.AsyncState; UpdateUserPhoto(userID, photo); } public delegate void UpdateUserPhotoDelegate(int userID, Binary photo); public void UpdateUserPhoto(int userID, Binary photo) { if (InvokeRequired) { var d = new UpdateUserPhotoDelegate(UpdateUserPhoto); Invoke(d, new object[] { userID, photo }); } else { if (photo != null) { var ms = new MemoryStream(photo.ToArray()); var bmp = new System.Drawing.Bitmap(ms); if (userID == theForm.AuthUserID) { pbMyPhoto.BackgroundImage = bmp; } else { pbPhoto.BackgroundImage = bmp; } } } }

  Read the article

• Initializing PHP class property declarations with simple expressions yields syntax error

  - by user171929

  According to the PHP docs, one can initialize properties in classes with the following restriction: "This declaration may include an initialization, but this initialization must be a constant value--that is, it must be able to be evaluated at compile time and must not depend on run-time information in order to be evaluated." I'm trying to initialize an array and having some issues. While this works fine: public $var = array( 1 => 4, 2 => 5, ); This creates a syntax error: public $var = array( 1 => 4, 2 => (4+1), ); Even this isn't accepted: public $var = 4+1; which suggests it's not a limitation of the array() language construct. Now, the last time I checked, "4+1" equated to a constant value that not only should be accepted, but should in fact be optimized away. In any case, it's certainly able to be evaluated at compile-time. So what's going on here? Is the limitation really along the lines of "cannot be any calculated expression at all", versus any expression "able to be evaluated at compile time"? The use of "evaluated" in the doc's language suggests that simple calculations are permitted, but alas.... If this is a bug in PHP, does anyone have a bug ID? I tried to find one but didn't have any luck.

  Read the article

• What are the fastest-performing options for a read-only, unordered collection of unique strings?

  - by Dan Tao

  Disclaimer: I realize the totally obvious answer to this question is HashSet<string>. It is absurdly fast, it is unordered, and its values are unique. But I'm just wondering, because HashSet<T> is a mutable class, so it has Add, Remove, etc.; and so I am not sure if the underlying data structure that makes these operations possible makes certain performance sacrifices when it comes to read operations -- in particular, I'm concerned with Contains. Basically, I'm wondering what are the absolute fastest-performing data structures in existence that can supply a Contains method for objects of type string. Within or outside of the .NET framework itself. I'm interested in all kinds of answers, regardless of their limitations. For example I can imagine that some structure might be restricted to strings of a certain length, or may be optimized depending on the problem domain (e.g., range of possible input values), etc. If it exists, I want to hear about it. One last thing: I'm not restricting this to read-only data structures. Obviously any read-write data structure could be embedded inside a read-only wrapper. The only reason I even mentioned the word "read-only" is that I don't have any requirement for a data structure to allow adding, removing, etc. If it has those functions, though, I won't complain.

  Read the article

• NSOperations or NSThread for bursts of smaller tasks that continuously cancel each other?

  - by RickiG

  Hi I would like to see if I can make a "search as you type" implementation, against a web service, that is optimized enough for it to run on an iPhone. The idea is that the user starts typing a word; "Foo", after each new letter I wait XXX ms. to see if they type another letter, if they don't, I call the web service using the word as a parameter. The web service call and the subsequent parsing of the result I would like to move to a different thread. I have written a simple SearchWebService class, it has only one public method: - (void) searchFor:(NSString*) str; This method tests if a search is already in progress (the user has had a XXX ms. delay in their typing) and subsequently stops that search and starts a new one. When a result is ready a delegate method is called: - (NSArray*) resultsReady; I can't figure out how to get this functionality 'threaded'. If I keep spawning new threads each time a user has a XXX ms. delay in the typing I end up in a bad spot with many threads, especially because I don't need any other search, but the last one. Instead of spawning threads continuously, I have tried keeping one thread running in the background all the time by: - (void) keepRunning { NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; SearchWebService *searchObj = [[SearchWebService alloc] init]; [[NSRunLoop currentRunLoop] run]; //keeps it alive [searchObj release]; [pool release]; } But I can't figure out how to access the "searchFor" method in the "searchObj" object, so the above code works and keeps running. I just can't message the searchObj or retrieve the resultReady objects? Hope someone could point me in the right direction, threading is giving me grief:) Thank you.

  Read the article

• Why would this Lua optimization hack help?

  - by Ian Boyd

  i'm looking over a document that describes various techniques to improve performance of Lua script code, and i'm shocked that such tricks would be required. (Although i'm quoting Lua, i've seen similar hacks in Javascript). Why would this optimization be required: For instance, the code for i = 1, 1000000 do local x = math.sin(i) end runs 30% slower than this one: local sin = math.sin for i = 1, 1000000 do local x = sin(i) end They're re-declaring sin function locally. Why would this be helpful? It's the job of the compiler to do that anyway. Why is the programmer having to do the compiler's job? i've seen similar things in Javascript; and so obviously there must be a very good reason why the interpreting compiler isn't doing its job. What is it? i see it repeatedly in the Lua environment i'm fiddling in; people redeclaring variables as local: local strfind = strfind local strlen = strlen local gsub = gsub local pairs = pairs local ipairs = ipairs local type = type local tinsert = tinsert local tremove = tremove local unpack = unpack local max = max local min = min local floor = floor local ceil = ceil local loadstring = loadstring local tostring = tostring local setmetatable = setmetatable local getmetatable = getmetatable local format = format local sin = math.sin What is going on here that people have to do the work of the compiler? Is the compiler confused by how to find format? Why is this an issue that a programmer has to deal with? Why would this not have been taken care of in 1993? i also seem to have hit a logical paradox: Optimizatin should not be done without profiling Lua has no ability to be profiled Lua should not be optimized

  Read the article

• iPhone: Using dispatch_after to mimick NSTimer

  - by Joseph Tura

  Don't know a whole lot about blocks. How would you go about mimicking a repeating NSTimer with dispatch_after? My problem is that I want to "pause" a timer when the app moves to the background, but subclassing NSTimer does not seem to work. I tried something which seems to work. I cannot judge its performance implications or whether it could be greatly optimized. Any input is welcome. #import "TimerWithPause.h" @implementation TimerWithPause @synthesize timeInterval; @synthesize userInfo; @synthesize invalid; @synthesize invocation; + (TimerWithPause *)scheduledTimerWithTimeInterval:(NSTimeInterval)aTimeInterval target:(id)aTarget selector:(SEL)aSelector userInfo:(id)aUserInfo repeats:(BOOL)aTimerRepeats { TimerWithPause *timer = [[[TimerWithPause alloc] init] autorelease]; timer.timeInterval = aTimeInterval; NSMethodSignature *signature = [[aTarget class] instanceMethodSignatureForSelector:aSelector]; NSInvocation *aInvocation = [NSInvocation invocationWithMethodSignature:signature]; [aInvocation setSelector:aSelector]; [aInvocation setTarget:aTarget]; [aInvocation setArgument:&timer atIndex:2]; timer.invocation = aInvocation; timer.userInfo = aUserInfo; if (!aTimerRepeats) { timer.invalid = YES; } [timer fireAfterDelay]; return timer; } - (void)fireAfterDelay { dispatch_time_t delay = dispatch_time(DISPATCH_TIME_NOW, self.timeInterval * NSEC_PER_SEC); dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0); dispatch_after(delay, queue, ^{ [invocation performSelectorOnMainThread:@selector(invoke) withObject:nil waitUntilDone:NO]; if (!invalid) { [self fireAfterDelay]; } }); } - (void)invalidate { invalid = YES; [invocation release]; invocation = nil; [userInfo release]; userInfo = nil; } - (void)dealloc { [self invalidate]; [super dealloc]; } @end

  Read the article

• why do we need advanced knowledge of mathematics & physics for programming?

  - by Sumeet

  Guys, I have been very good in mathematics and physics in my schools and colleges. Right now I am a programmer. Even in the colleges I have to engrossed my self into computers and programming things all the time. As I used to like it very much. But I have always felt the lack of advanced mathematics and physics in all the work I have done (Programs). Programming never asked me any advanced mathematics and physics knowledge in what I was very good. It always ask u some optimized loops, and different programming technologies which has never been covered in advanced mathematics and physics. Even at the time of selection in big College , such a kind of advanced knowledge is required. Time by time I got out of touch of all that facts and concepts (advanced mathematics and physics). And now after, 5 years in job I found it hard to resolve Differentiations and integrations from Trigonometry. Which sometimes make me feel like I have wasted time in those concepts because they are never used. (At that time I knew that I am going to be a programmer) If one need to be a programmer why do all this advanced knowledge is required. One can go with elementry knowledge a bit more. You never got to think like scientists and R&D person in your Schols and colleges for being a programmer? Just think and let me know your thoughts. I must be wrong somewhere in what I think , but not able to figure that out..? Regards Sumeet

  Read the article

• Which MySQL Fork/Version to Pick??

  - by Drew

  As most of you know, Sun acquired MySQL (and later Oracle acquired Sun), and during these acquisitions, there were a lot of FUD in MySQL community which resulted in creation of various forks. Today we have MySQL from MySQL, Percona (XtraDB) MySQL, OurDelta MySQL, MariaDB, Drizzle to name a few. Which brings us to the source of the problem. We are in the process of upgrading our databases (hardware/software) and I would like to know which one of the forks should I go with. Each has their own set of pros/cons. We are currently using MySQL 5.0.x from MySQL/Linux on an 8-core machine. Our new hardware is a monster with 32 cores and 32GB of memory connecting to a fast NetApp Storage via FC. I would like to stick with MySQL from MySQL but I have heard horror stories on how badly MySQL 5.1 performs on many cores. I have also heard that MySQL 5.4 performs better on multi-core machines but that's still not production ready. In addition, I have also heard a lot of good things about Percona builds. This is what I know so far: MySQL 5.1 from MySQL: Reliable choice, but doesn't scale well on a big machine Percona: Scales well, good backing company. I don't have much experience with it MariaDB: Don't know much about it besides that it was founded by Original MySQL developers (including Monty) OurDelta: Don't know much Drizzle: Mostly optimized for cloud computing I would like to know what's the general notion about this problem. Which build/version should I go with? How are you guys picking your builds/versions? Thanks!

  Read the article

• How do I optimize this postfix expression tree for speed?

  - by Peter Stewart

  Thanks to the help I received in this post: I have a nice, concise recursive function to traverse a tree in postfix order: deque <char*> d; void Node::postfix() { if (left != __nullptr) { left->postfix(); } if (right != __nullptr) { right->postfix(); } d.push_front(cargo); return; }; This is an expression tree. The branch nodes are operators randomly selected from an array, and the leaf nodes are values or the variable 'x', also randomly selected from an array. char *values[10]={"1.0","2.0","3.0","4.0","5.0","6.0","7.0","8.0","9.0","x"}; char *ops[4]={"+","-","*","/"}; As this will be called billions of times during a run of the genetic algorithm of which it is a part, I'd like to optimize it for speed. I have a number of questions on this topic which I will ask in separate postings. The first is: how can I get access to each 'cargo' as it is found. That is: instead of pushing 'cargo' onto a deque, and then processing the deque to get the value, I'd like to start processing it right away. I don't yet know about parallel processing in c++, but this would ideally be done concurrently on two different processors. In python, I'd make the function a generator and access succeeding 'cargo's using .next(). But I'm using c++ to speed up the python implementation. I'm thinking that this kind of tree has been around for a long time, and somebody has probably optimized it already. Any Ideas? Thanks

  Read the article

• LINQ Joins - Performance

  - by Meiscooldude

  I am curious on how exactly LINQ (not LINQ to SQL) is performing is joins behind the scenes in relation to how Sql Server performs joins. Sql Server before executing a query, generates an Execution Plan. The Execution Plan is basically an Expression Tree on what it believes is the best way to execute the query. Each node provides information on whether to do a Sort, Scan, Select, Join, ect. On a 'Join' node in our execution plan, we can see three possible algorithms; Hash Join, Merge Join, and Nested Loops Join. Sql Server will choose which algorithm to for each Join operation based on expected number of rows in Inner and Outer tables, what type of join we are doing (some algorithms don't support all types of joins), whether we need data ordered, and probably many other factors. Join Algorithms: Nested Loop Join: Best for small inputs, can be optimized with ordered inner table. Merge Join: Best for medium to large inputs sorted inputs, or an output that needs to be ordered. Hash Join: Best for medium to large inputs, can be parallelized to scale linearly. LINQ Query: DataTable firstTable, secondTable; ... var rows = from firstRow in firstTable.AsEnumerable () join secondRow in secondTable.AsEnumerable () on firstRow.Field<object> (randomObject.Property) equals secondRow.Field<object> (randomObject.Property) select new {firstRow, secondRow}; SQL Query: SELECT * FROM firstTable fT INNER JOIN secondTable sT ON fT.Property = sT.Property Sql Server might use a Nested Loop Join if it knows there are a small number of rows from each table, a merge join if it knows one of the tables has an index, and Hash join if it knows there are a lot of rows on either table and neither has an index. Does Linq choose its algorithm for joins? or does it always use one?

  Read the article

• array of structures, or structure of arrays?

  - by Jason S

  Hmmm. I have a table which is an array of structures I need to store in Java. The naive don't-worry-about-memory approach says do this: public class Record { final private int field1; final private int field2; final private long field3; /* constructor & accessors here */ } List<Record> records = new ArrayList<Record>(); If I end up using a large number ( 106 ) of records, where individual records are accessed occasionally, one at a time, how would I figure out how the preceding approach (an ArrayList) would compare with an optimized approach for storage costs: public class OptimizedRecordStore { final private int[] field1; final private int[] field2; final private long[] field3; Record getRecord(int i) { return new Record(field1[i],field2[i],field3[i]); } /* constructor and other accessors & methods */ } edit: assume the # of records is something that is changed infrequently or never I'm probably not going to use the OptimizedRecordStore approach, but I want to understand the storage cost issue so I can make that decision with confidence. obviously if I add/change the # of records in the OptimizedRecordStore approach above, I either have to replace the whole object with a new one, or remove the "final" keyword. kd304 brings up a good point that was in the back of my mind. In other situations similar to this, I need column access on the records, e.g. if field1 and field2 are "time" and "position", and it's important for me to get those values as an array for use with MATLAB, so I can graph/analyze them efficiently.

  Read the article

• Android Cursor strange behaviour

  - by sandis

  After many houres of bug searching in a big app, I have finally tracked down the bug. This logging captures the problem: Log.d(TAG,"buildList, DBresult.getInt(1): "+DBresult.getInt(1)); Log.d(TAG,"buildList, DBresult.getString(1): "+DBresult.getString(1)); Log.d(TAG,"buildList, DBresult.getInt(4): "+DBresult.getInt(4)); Log.d(TAG,"buildList, DBresult.getString(4): "+DBresult.getString(4)); The resulting output: 05-06 11:11:20.123: DEBUG/TodoList(18943): buildList, DBresult.getInt(1): 0 05-06 11:11:20.123: DEBUG/TodoList(18943): buildList, DBresult.getString(1): false 05-06 11:11:20.123: DEBUG/TodoList(18943): buildList, DBresult.getInt(4): 0 05-06 11:11:20.123: DEBUG/TodoList(18943): buildList, DBresult.getString(4): true There are no backgroung threads running. As you can see the problem is that '0' is interpreted as false on one occasion, and as true on another. Since I am completely lost on how this can happen, I dont know how to proceed. What could possibly result in such a behaviour? Both the columns are of the type "boolean", i.e a numeric in sqlite. Unfortunately the string returned is the correct value, while the integer is always 0. If I export the database to my computer and check it with SQlite Administrator I can see that the values are correctly set, it is only the getInt()-function that always returns 0. I know for a fact that this works in other apps I have coded, and I dont know why this has stopped working. I have tried compiling the code under 2.0, 2.0.1 and 2.1, and it always appears. I can make my app runnable again by getting boolean values like this: myBool= (DBresult.getString(0).equals("true")); but that is both ugly and not optimized. Any suggestions on what is causing this behaviour is welcome. Cheers,

  Read the article

• Tail recursion and memoization with C#

  - by Jay

  I'm writing a function that finds the full path of a directory based on a database table of entries. Each record contains a key, the directory's name, and the key of the parent directory (it's the Directory table in an MSI if you're familiar). I had an iterative solution, but it started looking a little nasty. I thought I could write an elegant tail recursive solution, but I'm not sure anymore. I'll show you my code and then explain the issues I'm facing. Dictionary<string, string> m_directoryKeyToFullPathDictionary = new Dictionary<string, string>(); ... private string ExpandDirectoryKey(Database database, string directoryKey) { // check for terminating condition string fullPath; if (m_directoryKeyToFullPathDictionary.TryGetValue(directoryKey, out fullPath)) { return fullPath; } // inductive step Record record = ExecuteQuery(database, "SELECT DefaultDir, Directory_Parent FROM Directory where Directory.Directory='{0}'", directoryKey); // null check string directoryName = record.GetString("DefaultDir"); string parentDirectoryKey = record.GetString("Directory_Parent"); return Path.Combine(ExpandDirectoryKey(database, parentDirectoryKey), directoryName); } This is how the code looked when I realized I had a problem (with some minor validation/massaging removed). I want to use memoization to short circuit whenever possible, but that requires me to make a function call to the dictionary to store the output of the recursive ExpandDirectoryKey call. I realize that I also have a Path.Combine call there, but I think that can be circumvented with a ... + Path.DirectorySeparatorChar + .... I thought about using a helper method that would memoize the directory and return the value so that I could call it like this at the end of the function above: return MemoizeHelper( m_directoryKeyToFullPathDictionary, Path.Combine(ExpandDirectoryKey(database, parentDirectoryKey)), directoryName); But I feel like that's cheating and not going to be optimized as tail recursion. Any ideas? Should I be using a completely different strategy? This doesn't need to be a super efficient algorithm at all, I'm just really curious. I'm using .NET 4.0, btw. Thanks!

  Read the article

• What hash algorithms are paralellizable? Optimizing the hashing of large files utilizing on mult-co

  - by DanO

  I'm interested in optimizing the hashing of some large files (optimizing wall clock time). The I/O has been optimized well enough already and the I/O device (local SSD) is only tapped at about 25% of capacity, while one of the CPU cores is completely maxed-out. I have more cores available, and in the future will likely have even more cores. So far I've only been able to tap into more cores if I happen to need multiple hashes of the same file, say an MD5 AND a SHA256 at the same time. I can use the same I/O stream to feed two or more hash algorithms, and I get the faster algorithms done for free (as far as wall clock time). As I understand most hash algorithms, each new bit changes the entire result, and it is inherently challenging/impossible to do in parallel. Are any of the mainstream hash algorithms parallelizable? Are there any non-mainstream hashes that are parallelizable (and that have at least a sample implementation available)? As future CPUs will trend toward more cores and a leveling off in clock speed, is there any way to improve the performance of file hashing? (other than liquid nitrogen cooled overclocking?) or is it inherently non-parallelizable?

  Read the article

• django dynamically deduce SITE_ID according to the domain

  - by dcrodjer

  I am trying to develop a site which will render multiple customized sites according to the domain name (subdomain to be more precise). My all the domain names are redirected to the So for each site there will be a corresponding model which defines how the site should look (SITE - SITE_SETTINGS) What will be the best way to utilize the django sites framework to get the SITE_ID of the current site from the domain name instead of hard-coding it in the settings files (django sites documentation) and run database queries, render the views accordingly? If using multiple settings file is my only option can this (wsgi script handle domain name) be done? Update So finally, following lukes answer, what I will do is define a custom middleware which makes the views available with the important vars required according to the domain. And as far as sitemaps and comments is concerned, I will have to customize sitemaps app and a custom sites model on which the other models of sites will be based. And since the comments system is based on the hard-coded sitemap ID I can use it just as is on the models (models will already be filtered according to the site based on my sites framework) though the permalink feature will have to be customized. So a lot of customization. Please suggest if I am going wrong anywhere in this because I have to ensure that the features of the project are optimized. Thanks!

  Read the article

• MySQL query puzzle - finding what WOULD have been the most recent date

  - by Hank

  I've looked all over and haven't yet found an intelligent way to handle this, though I feel sure one is possible: One table of historical data has quarterly information: CREATE TABLE Quarterly ( unique_ID INT UNSIGNED NOT NULL, date_posted DATE NOT NULL, datasource TINYINT UNSIGNED NOT NULL, data FLOAT NOT NULL, PRIMARY KEY (unique_ID)); Another table of historical data (which is very large) contains daily information: CREATE TABLE Daily ( unique_ID INT UNSIGNED NOT NULL, date_posted DATE NOT NULL, datasource TINYINT UNSIGNED NOT NULL, data FLOAT NOT NULL, qtr_ID INT UNSIGNED, PRIMARY KEY (unique_ID)); The qtr_ID field is not part of the feed of daily data that populated the database - instead, I need to retroactively populate the qtr_ID field in the Daily table with the Quarterly.unique_ID row ID, using what would have been the most recent quarterly data on that Daily.date_posted for that data source. For example, if the quarterly data is 101 2009-03-31 1 4.5 102 2009-06-30 1 4.4 103 2009-03-31 2 7.6 104 2009-06-30 2 7.7 105 2009-09-30 1 4.7 and the daily data is 1001 2009-07-14 1 3.5 ?? 1002 2009-07-15 1 3.4 && 1003 2009-07-14 2 2.3 ^^ then we would want the ?? qtr_ID field to be assigned '102' as the most recent quarter for that data source on that date, and && would also be '102', and ^^ would be '104'. The challenges include that both tables (particularly the daily table) are actually very large, they can't be normalized to get rid of the repetitive dates or otherwise optimized, and for certain daily entries there is no preceding quarterly entry. I have tried a variety of joins, using datediff (where the challenge is finding the minimum value of datediff greater than zero), and other attempts but nothing is working for me - usually my syntax is breaking somewhere. Any ideas welcome - I'll execute any basic ideas or concepts and report back.

  Read the article

• changing the serialization procedure for a graph of objects (.net framework)

  - by pierusch

  Hello I'm developing a scientific application using .net framework. The application depends heavily upon a large data structure (a tree like structure) that has been serialized using a standard binaryformatter object. The graph structure looks like this: <serializable()>Public class BigObjet inherits list(of smallObject) end class <serializable()>public class smallObject inherits list(of otherSmallerObjects) end class ... The binaryFormatter object does a nice job but it's not optimized at all and the entire data structure reaches around 100Mb on my filesystem. Deserialization works too but it's pretty slow (around 30seconds on my quad core). I've found a nice .dll on codeproject (see "optimizing serialization...") so I wrote a modified version of the classes above overriding the default serialization/deserialization procedure reaching very good results. The problem is this: I can't lose the data previosly serialized with the old version and I'd like to be able to use the new serialization/deserialization method. I have some ideas but I'm pretty sure someone will be able to give me a proper and better advice ! use an "helper" graph of objects who takes care of the entire serialization/deserialization procedure reading data from the old format and converting them into the classes I nedd. This could work but the binaryformatter "needs" to know the types being serialized so........ :( modify the "old" graph to include a modified version of serialization procedure...so I'll be able to deserialize old file and save them with the new format......this doesn't sound too good imho. well any help will be higly highly appreciated :)

  Read the article

< Previous Page | 159 160 161 162 163 164 165 166 167 168 169 170  | Next Page >