Search Results

Search found 7672 results on 307 pages for 'compiler optimization'.

Page 75/307 | < Previous Page | 71 72 73 74 75 76 77 78 79 80 81 82  | Next Page >

• Java: ArrayList bottleneck

  - by Jack

  Hello, while profiling a java application that calculates hierarchical clustering of thousands of elements I realized that ArrayList.get occupies like half of the CPU needed in the clusterization part of the execution. The algorithm searches the two more similar elements (so it is O(n*(n+1)/2) ), here's the pseudo code: int currentMax = 0.0f for (int i = 0 to n) for (int j = i to n) get content i-th and j-th if their similarity > currentMax update currentMax merge the two clusters So effectively there are a lot of ArrayList.get involved. Is there a faster way? I though that since ArrayList should be a linear array of references it should be the quickest way and maybe I can't do anything since there are simple too many gets.. but maybe I'm wrong. I don't think using a HashMap could work since I need to get them all on every iteration and map.values() should be backed by an ArrayList anyway.. Otherwise should I try other collection libraries that are more optimized? Like google's one, or apache one.. Thanks

  Read the article

• MySQL Query Select using sub-select takes too long

  - by True Soft

  I noticed something strange while executing a select from 2 tables: SELECT * FROM table_1 WHERE id IN ( SELECT id_element FROM table_2 WHERE column_2=3103); This query took approximatively 242 seconds. But when I executed the subquery SELECT id_element FROM table_2 WHERE column_2=3103 it took less than 0.002s (and resulted 2 rows). Then, when I did SELECT * FROM table_1 WHERE id IN (/* prev.result */) it was the same: 0.002s. I was wondering why MySQL is doing the first query like that, taking much more time than the last 2 queries separately? Is it an optimal solution for selecting something based from the results of a sub-query? Other details: table_1 has approx. 9000 rows, and table_2 has 90000 rows. After I added an index on column_2 from table_2, the first query took 0.15s.

  Read the article

• Optimizations employed by ORM's

  - by Kartoch

  I'm teaching JEE, especially JPA, Spring and Spring MVC. As I have not so much experience in large projects, it is difficult to know what to present to students about optimisation of ORM. At the present time, I present some classic optimisation tricks: prepared statements (most of ORM implicitely uses it by default) first and second-level caches "write first, optimize later" it is possible to switch off ORM and send SQL commands directly to the database for very frequent, specialized and costly requests Is there any other point the community see about other way to optimize ORM ? I'm especially interested by DAO patterns...

  Read the article

• Xcode 4 and cocos2D 1.0.0 beta Uncategorized errors and Info.plist doesn't exist

  - by badben

  I just installed the xcode 4 sdk and the cocos2d 1.0.0 beta template. I just created a new project with the cocos2d template. But when I build I got these errors : (for information my previous projects developed with xcode 3 have the same problem) warning: couldn't add 'com.apple.XcodeGenerated' tag to '/Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build': Error Domain=NSPOSIXErrorDomain Code=2 UserInfo=0x201dde680 "The operation couldn’t be completed. No such file or directory" error: unable to create '/Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates' (Permission denied) error: unable to create '/Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Products' (Permission denied) Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build/Debug-iphonesimulator/xcode4.build/Objects-normal/i386 Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/PrecompiledHeaders/Prefix-dflnzjtztxdgjwhistrvvjxetfrg Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build/Debug-iphonesimulator/xcode4.build Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build/Debug-iphonesimulator/xcode4.build Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build/Debug-iphonesimulator/xcode4.build Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build/Debug-iphonesimulator/xcode4.build Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/PrecompiledHeaders/Prefix-fqemzerugrwojibbegzkffljkxqs Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Intermediates/xcode4.build/Debug-iphonesimulator/xcode4.build Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Index/PrecompiledHeaders/Prefix-dbtcglhksokwygezixirqkgfipsr_ast Unable to create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Index/PrecompiledHeaders/Prefix-gdirtpasdqzasnclnkzguimarjpd_ast error: couldn't create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Products/Debug-iphonesimulator/xcode4.app: Permission denied error: couldn't create directory /Users/Benoit/Library/Developer/Xcode/DerivedData/xcode4-bswxazfuwbsguiasyatbtlmvbpps/Build/Products/Debug-iphonesimulator/xcode4.app: Permission denied The file “Info.plist” doesn’t exist. Please help !!

  Read the article

• Determining the maximum stack depth

  - by Joa Ebert

  Imagine I have a stack-based toy language that comes with the operations Push, Pop, Jump and If. I have a program and its input is the toy language. For instance I get the sequence Push 1 Push 1 Pop Pop In that case the maximum stack would be 2. A more complicated example would use branches. Push 1 Push true If .success Pop Jump .continue .success: Push 1 Push 1 Pop Pop Pop .continue: In this case the maximum stack would be 3. However it is not possible to get the maximum stack by walking top to bottom as shown in this case since it would result in a stack-underflow error actually. CFGs to the rescue you can build a graph and walk every possible path of the basic blocks you have. However since the number of paths can grow quickly for n vertices you get (n-1)! possible paths. My current approach is to simplify the graph as much as possible and to have less possible paths. This works but I would consider it ugly. Is there a better (read: faster) way to attack this problem? I am fine if the algorithm produces a stack depth that is not optimal. If the correct stack size is m then my only constraint is that the result n is n = m. Is there maybe a greedy algorithm available that would produce a good result here?

  Read the article

• Shouldn't prepared statements be much more fsater?

  - by silversky

  $s = explode (" ", microtime()); $s = $s[0]+$s[1]; $con = mysqli_connect ('localhost', 'test', 'pass', 'db') or die('Err'); for ($i=0; $i<1000; $i++) { $stmt = $con -> prepare( " SELECT MAX(id) AS max_id , MIN(id) AS min_id FROM tb "); $stmt -> execute(); $stmt->bind_result($M,$m); $stmt->free_result(); $rand = mt_rand( $m , $M ).'<br/>'; $res = $con -> prepare( " SELECT * FROM tb WHERE id >= ? LIMIT 0,1 "); $res -> bind_param("s", $rand); $res -> execute(); $res->free_result(); } $e = explode (" ", microtime()); $e = $e[0]+$e[1]; echo number_format($e-$s, 4, '.', ''); // and: $link = mysql_connect ("localhost", "test", "pass") or die (); mysql_select_db ("db") or die ("Unable to select database".mysql_error()); for ($i=0; $i<1000; $i++) { $range_result = mysql_query( " SELECT MAX(`id`) AS max_id , MIN(`id`) AS min_id FROM tb "); $range_row = mysql_fetch_object( $range_result ); $random = mt_rand( $range_row->min_id , $range_row->max_id ); $result = mysql_query( " SELECT * FROM tb WHERE id >= $random LIMIT 0,1 "); } defenitly prepared statements are much more safer but also every where it says that they are much faster BUT in my test on the above code I have: - 2.45 sec for prepared statements - 5.05 sec for the secon example What do you think I'm doing wrong? Should I use the second solution or I should try to optimize the prep stmt?

  Read the article

• How to overcome vc++ warning C4003 while writing common code for both gcc and vc++

  - by compbugs

  I have a code that is compiled in both gcc and vc++. The code has a common macro which is called in two scenarios. When we pass some parameters to it. When we don't want to pass any parameters to it. An example of such a code is: #define B(X) A1##X int main() { int B(123), B(); return 0; } The expect output from the pre-processing step of compilation is: int main() { int A1123, A1; return 0; } The output for both gcc and vc++ is as expected, but vc++ gives a warning: warning C4003: not enough actual parameters for macro 'B' How can I remove this warning and yet get the expected output? Thanks.

  Read the article

• Question about Objective C calling convention and argument passing on ARM

  - by overboming

  I want to know how objective C runtime handle arguments when I call a objective C method like [NSString stringWithFomat:@"%@, %@", @"Hello", @"World"] There are three arguments for this objective C call, how does it work compared to typical way on a ARM system. I have known register r0, r1, r2, r3 will hold first 4 arguments, how about there are additional arguments? How does it put them on a stack and pop them later?

  Read the article

• Linker error when compiling boost.asio example

  - by Alon

  Hi, I'm trying to learn a little bit C++ and Boost.Asio. I'm trying to compile the following code example: #include <iostream> #include <boost/array.hpp> #include <boost/asio.hpp> using boost::asio::ip::tcp; int main(int argc, char* argv[]) { try { if (argc != 2) { std::cerr << "Usage: client <host>" << std::endl; return 1; } boost::asio::io_service io_service; tcp::resolver resolver(io_service); tcp::resolver::query query(argv[1], "daytime"); tcp::resolver::iterator endpoint_iterator = resolver.resolve(query); tcp::resolver::iterator end; tcp::socket socket(io_service); boost::system::error_code error = boost::asio::error::host_not_found; while (error && endpoint_iterator != end) { socket.close(); socket.connect(*endpoint_iterator++, error); } if (error) throw boost::system::system_error(error); for (;;) { boost::array<char, 128> buf; boost::system::error_code error; size_t len = socket.read_some(boost::asio::buffer(buf), error); if (error == boost::asio::error::eof) break; // Connection closed cleanly by peer. else if (error) throw boost::system::system_error(error); // Some other error. std::cout.write(buf.data(), len); } } catch (std::exception& e) { std::cerr << e.what() << std::endl; } return 0; } With the following command line: g++ -I /usr/local/boost_1_42_0 a.cpp and it throws an unclear error: /tmp/ccCv9ZJA.o: In function `__static_initialization_and_destruction_0(int, int)': a.cpp:(.text+0x654): undefined reference to `boost::system::get_system_category()' a.cpp:(.text+0x65e): undefined reference to `boost::system::get_generic_category()' a.cpp:(.text+0x668): undefined reference to `boost::system::get_generic_category()' a.cpp:(.text+0x672): undefined reference to `boost::system::get_generic_category()' a.cpp:(.text+0x67c): undefined reference to `boost::system::get_system_category()' /tmp/ccCv9ZJA.o: In function `boost::system::error_code::error_code()': a.cpp:(.text._ZN5boost6system10error_codeC2Ev[_ZN5boost6system10error_codeC5Ev]+0x10): undefined reference to `boost::system::get_system_category()' /tmp/ccCv9ZJA.o: In function `boost::asio::error::get_system_category()': a.cpp:(.text._ZN5boost4asio5error19get_system_categoryEv[boost::asio::error::get_system_category()]+0x7): undefined reference to `boost::system::get_system_category()' /tmp/ccCv9ZJA.o: In function `boost::asio::detail::posix_thread::~posix_thread()': a.cpp:(.text._ZN5boost4asio6detail12posix_threadD2Ev[_ZN5boost4asio6detail12posix_threadD5Ev]+0x1d): undefined reference to `pthread_detach' /tmp/ccCv9ZJA.o: In function `boost::asio::detail::posix_thread::join()': a.cpp:(.text._ZN5boost4asio6detail12posix_thread4joinEv[boost::asio::detail::posix_thread::join()]+0x25): undefined reference to `pthread_join' /tmp/ccCv9ZJA.o: In function `boost::asio::detail::posix_tss_ptr<boost::asio::detail::call_stack<boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false> > >::context>::~posix_tss_ptr()': a.cpp:(.text._ZN5boost4asio6detail13posix_tss_ptrINS1_10call_stackINS1_15task_io_serviceINS1_13epoll_reactorILb0EEEEEE7contextEED2Ev[_ZN5boost4asio6detail13posix_tss_ptrINS1_10call_stackINS1_15task_io_serviceINS1_13epoll_reactorILb0EEEEEE7contextEED5Ev]+0xf): undefined reference to `pthread_key_delete' /tmp/ccCv9ZJA.o: In function `boost::asio::detail::posix_tss_ptr<boost::asio::detail::call_stack<boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false> > >::context>::posix_tss_ptr()': a.cpp:(.text._ZN5boost4asio6detail13posix_tss_ptrINS1_10call_stackINS1_15task_io_serviceINS1_13epoll_reactorILb0EEEEEE7contextEEC2Ev[_ZN5boost4asio6detail13posix_tss_ptrINS1_10call_stackINS1_15task_io_serviceINS1_13epoll_reactorILb0EEEEEE7contextEEC5Ev]+0x22): undefined reference to `pthread_key_create' collect2: ld returned 1 exit status How can I fix it? Thank you.

  Read the article

• PHP Flush: How Often and Best Practises

  - by Cory Dee

  I just finished reading this post: http://developer.yahoo.com/performance/rules.html#flush and have already implemented a flush after the top portion of my page loads (head, css, top banner/search/nav). Is there any performance hit in flushing? Is there such a thing as doing it too often? What are the best practices? If I am going to hit an external API for data, would it make sense to flush before hand so that the user isn't waiting on that data to come back, and can at least get some data before hand? Thanks to everyone in advance.

  Read the article

• Fastest way to remove non-numeric characters from a VARCHAR in SQL Server

  - by Dan Herbert

  I'm writing an import utility that is using phone numbers as a unique key within the import. I need to check that the phone number does not already exist in my DB. The problem is that phone numbers in the DB could have things like dashes and parenthesis and possibly other things. I wrote a function to remove these things, the problem is that it is slow and with thousands of records in my DB and thousands of records to import at once, this process can be unacceptably slow. I've already made the phone number column an index. I tried using the script from this post: http://stackoverflow.com/questions/52315/t-sql-trim-nbsp-and-other-non-alphanumeric-characters But that didn't speed it up any. Is there a faster way to remove non-numeric characters? Something that can perform well when 10,000 to 100,000 records have to be compared. Whatever is done needs to perform fast. Update Given what people responded with, I think I'm going to have to clean the fields before I run the import utility. To answer the question of what I'm writing the import utility in, it is a C# app. I'm comparing BIGINT to BIGINT now, with no need to alter DB data and I'm still taking a performance hit with a very small set of data (about 2000 records). Could comparing BIGINT to BIGINT be slowing things down? I've optimized the code side of my app as much as I can (removed regexes, removed unneccessary DB calls). Although I can't isolate SQL as the source of the problem anymore, I still feel like it is.

  Read the article

• graphics programming

  - by PVB

  hi , i would like to program some graphic figures such as line, circle,etc. i have used turboc++ 3.0 for dos graphics. i would like to do the same with the compilers dev c++ or code blocks or vc++. i would like to implement dda and bresenhems line and circle drawing algorithm. how should i go about implementing these programs through these compilers (not the command line tools). i have a really vague picture of graphics programming.please help.. thanks in advance. please note : i have a nvidia graphics card 1gb.. so i cannot use dos graphics (i think the card is the reason).

  Read the article

• Error starting an Android program.

  - by Gabriel A. Zorrilla

  [2010-04-16 23:31:34 - MobileDataKeeper] Error in an XML file: aborting build. [2010-04-16 23:31:57 - MobileDataKeeper] Error in an XML file: aborting build. [2010-04-16 23:32:28 - MobileDataKeeper] Error in an XML file: aborting build. [2010-04-16 23:34:56 - MobileDataKeeper] Error in an XML file: aborting build. [2010-04-16 23:35:14 - MobileDataKeeper] res/layout/main.out.xml:0: error: Resource entry main is already defined. [2010-04-16 23:35:14 - MobileDataKeeper] res/layout/main.out.out.xml:0: Originally defined here. [2010-04-16 23:35:14 - MobileDataKeeper] res/layout/main.xml:0: error: Resource entry main is already defined. [2010-04-16 23:35:14 - MobileDataKeeper] res/layout/main.out.out.xml:0: Originally defined here. [2010-04-16 23:35:14 - MobileDataKeeper] /media/Mis Documentos/Dropbox/Eclipse/MobileDataKeeper/res/layout/main.out.out.xml:1: error: Error parsing XML: no element found [2010-04-16 23:35:14 - MobileDataKeeper] /media/Mis Documentos/Dropbox/Eclipse/MobileDataKeeper/res/layout/main.out.xml:1: error: Error parsing XML: no element found [2010-04-16 23:35:49 - MobileDataKeeper] Error in an XML file: aborting build. Main.xml.out.* are empty. The MobileDataKeeper.java is default and the Mainx.xml out is: <?xml version="1.0" encoding="utf-8"?> <RelativeLayout android:id="@+id/RelativeLayout01" android:layout_width="fill_parent" android:layout_height="fill_parent" xmlns:android="http://schemas.android.com/apk/res/android" /> Every time i change something in the Main.xml get this problem. It's very frustrating and I don't know what the heck is going on. A trained eye is requiered! Thanks!

  Read the article

• Tips on creating user interfaces and optimizing the user experience

  - by Saif Bechan

  I am currently working on a project where a lot of user interaction is going to take place. There is also a commercial side as people can buy certain items and services. In my opinion a good blend of user interface, speed and security is essential for these types of websites. It is fairly easy to use ajax and JavaScript nowadays to do almost everything, as there are a lot of libraries available such as jQuery and others. But this can have some performance and incompatibility issues. This can lead to users just going to the next website. The overall look of the website is important too. Where to place certain buttons, where to place certain types of articles such as faq and support. Where and how to display error messages so that the user sees them but are not bothering him. And an overall color scheme is important too. The basic question is: How to create an interface that triggers a user to buy/use your services I know psychology also plays a huge role in how users interact with your website. The color scheme for example is important. When the colors are irritating on a website you just want to click away. I have not found any articles that explain those concept. Does anyone have any tips and/or recourses where i can get some articles that guide you in making the correct choices for your website.

  Read the article

• ILOG CPLEX: how to populate IloLPMatrix while using addGe to set up the model?

  - by downer

  I have a queatoin about IloLPMatrix and addGe. I was trying to follow the example of AdMIPex5.java to generate user defined cutting planes based on the solution to the LP relaxation. The difference is that eh initial MIP model is not read in from a mps file, but set up in the code using methods like addGe, addLe etc. I think this is why I ran into problems while copying the exampe to do the following. IloLPMatrix lp = (IloLPMatrix)cplex.LPMatrixIterator().next(); lp from the above line turns to be NULL. I am wondering 1. What is the relationship between IloLPMatrix and the addLe, addGe commands? I tried to addLPMatrix() to the model, and then used model.addGe methods. but the LPMatrix seems to be empty still. How do I populate the IloLPMatrix of the moel according to the value that I had set up using addGe and addLe. Is the a method to this easily, or do I have to set them up row by row myself? I was doing this to get the number of variables and their values by doing lp.getNumVars(). Is there other methods that I can use to get the number of variables and their values wihout doing these, since my system is set up by addLe, addGe etc? Thanks a lot for your help on this.

  Read the article

• vectorizing a for loop in numpy/scipy?

  - by user248237

  I'm trying to vectorize a for loop that I have inside of a class method. The for loop has the following form: it iterates through a bunch of points and depending on whether a certain variable (called "self.condition_met" below) is true, calls a pair of functions on the point, and adds the result to a list. Each point here is an element in a vector of lists, i.e. a data structure that looks like array([[1,2,3], [4,5,6], ...]). Here is the problematic function: def myClass: def my_inefficient_method(self): final_vector = [] # Assume 'my_vector' and 'my_other_vector' are defined numpy arrays for point in all_points: if not self.condition_met: a = self.my_func1(point, my_vector) b = self.my_func2(point, my_other_vector) else: a = self.my_func3(point, my_vector) b = self.my_func4(point, my_other_vector) c = a + b final_vector.append(c) # Choose random element from resulting vector 'final_vector' self.condition_met is set before my_inefficient_method is called, so it seems unnecessary to check it each time, but I am not sure how to better write this. Since there are no destructive operations here it is seems like I could rewrite this entire thing as a vectorized operation -- is that possible? any ideas how to do this?

  Read the article

• Translate a<b to IR Trees

  - by drozzy

  I have to translate the mini-java (java like language) statements into intermediate-representation trees. But for this question I have no idea what it is asking... a>b moves a 1 or 0 into some newly defined temporary, and whose right-hand side is a temporary Does the wording make sense to anyone? (I am using the Java compilers book, and it is question 7.2d) in ch7.)

  Read the article

• How does Visual Studio decide the order in which stack variables should be allocated?

  - by Jason

  I'm trying to turn some of the programs in gera's Insecure Programming by example into client/server applications that could be used in capture the flag scenarios to teach exploit development. The problem I'm having is that I'm not sure how Visual Studio (I'm using 2005 Professional Edition) decides where to allocate variables on the stack. When I compile and run example 1: int main() { int cookie; char buf[80]; printf("buf: %08x cookie: %08x\n", &buf, &cookie); gets(buf); if (cookie == 0x41424344) printf("you win!\n"); } I get the following result: buf: 0012ff14 cookie: 0012ff64 buf starts at an address eighty bytes lower than cookie, and any four bytes that are copied in buf after the first eighty will appear in cookie. The problem I'm having is when I place this code in some other function. When I compile and run the following code, I get a different result: buf appears at an address greater than cookie's. void ClientSocketHandler(SOCKET cs){ int cookie; char buf[80]; char stringToSend[160]; int numBytesRecved; int totalNumBytes; sprintf(stringToSend,"buf: %08x cookie: %08x\n",&buf,&cookie); send(cs,stringToSend,strlen(stringToSend),NULL); The result is: buf: 0012fd00 cookie: 0012fcfc Now there is no way to set cookie to arbitrary data via overwriting buf. Is there any way to tell Visual Studio to allocate cookie before buf? Is there any way to tell beforehand how the variables will be allocated? Thanks, Jason

  Read the article

• SQLite3-ruby extremely slow under 1.9.1?

  - by NilObject

  I decided to upgrade my server to Ruby 1.9.1, and a lot of things are indeed much faster. However, I have a process that dumps a database to sqlite, and it's become glacially slow. What used to take 30 seconds now takes upwards of 10 minutes. The code does several create table statements, and then lots of inserts. The insert statements nearly all use placeholders (?), so SQLite is doing the heavy lifting of binding the parameters. In short, I can't see why this particular usage has slowed down so much. Does anyone know of any problems that have caused it? I'm using sqlite3-ruby (1.2.5), and I'm hoping that someone has encountered this and profiled it. If not, I guess I'm going to learn how to profile ruby code :)

  Read the article

• Compiling NyARToolKit on iPhone 3.0

  - by Vivas

  Hi, Has anyone here succeeded in getting NyARToolKit to run on iPhone 3.0? I'm fighting with this but I am at least down to a couple of errors: Duplicate symbol _spriteVertices (referring to the 3DGraphicsView & NyARToolKitCrossCompileAppDelegate). glView.delegate = self; (error in NyARToolKitCrossCompileAppDelegate.m) Any suggestions would be greatly appreciated.

  Read the article

• How to optimize my PageRank calculation?

  - by asmaier

  In the book Programming Collective Intelligence I found the following function to compute the PageRank: def calculatepagerank(self,iterations=20): # clear out the current PageRank tables self.con.execute("drop table if exists pagerank") self.con.execute("create table pagerank(urlid primary key,score)") self.con.execute("create index prankidx on pagerank(urlid)") # initialize every url with a PageRank of 1.0 self.con.execute("insert into pagerank select rowid,1.0 from urllist") self.dbcommit() for i in range(iterations): print "Iteration %d" % i for (urlid,) in self.con.execute("select rowid from urllist"): pr=0.15 # Loop through all the pages that link to this one for (linker,) in self.con.execute("select distinct fromid from link where toid=%d" % urlid): # Get the PageRank of the linker linkingpr=self.con.execute("select score from pagerank where urlid=%d" % linker).fetchone()[0] # Get the total number of links from the linker linkingcount=self.con.execute("select count(*) from link where fromid=%d" % linker).fetchone()[0] pr+=0.85*(linkingpr/linkingcount) self.con.execute("update pagerank set score=%f where urlid=%d" % (pr,urlid)) self.dbcommit() However, this function is very slow, because of all the SQL queries in every iteration >>> import cProfile >>> cProfile.run("crawler.calculatepagerank()") 2262510 function calls in 136.006 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 136.006 136.006 <string>:1(<module>) 1 20.826 20.826 136.006 136.006 searchengine.py:179(calculatepagerank) 21 0.000 0.000 0.528 0.025 searchengine.py:27(dbcommit) 21 0.528 0.025 0.528 0.025 {method 'commit' of 'sqlite3.Connecti 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler 1339864 112.602 0.000 112.602 0.000 {method 'execute' of 'sqlite3.Connec 922600 2.050 0.000 2.050 0.000 {method 'fetchone' of 'sqlite3.Cursor' 1 0.000 0.000 0.000 0.000 {range} So I optimized the function and came up with this: def calculatepagerank2(self,iterations=20): # clear out the current PageRank tables self.con.execute("drop table if exists pagerank") self.con.execute("create table pagerank(urlid primary key,score)") self.con.execute("create index prankidx on pagerank(urlid)") # initialize every url with a PageRank of 1.0 self.con.execute("insert into pagerank select rowid,1.0 from urllist") self.dbcommit() inlinks={} numoutlinks={} pagerank={} for (urlid,) in self.con.execute("select rowid from urllist"): inlinks[urlid]=[] numoutlinks[urlid]=0 # Initialize pagerank vector with 1.0 pagerank[urlid]=1.0 # Loop through all the pages that link to this one for (inlink,) in self.con.execute("select distinct fromid from link where toid=%d" % urlid): inlinks[urlid].append(inlink) # get number of outgoing links from a page numoutlinks[urlid]=self.con.execute("select count(*) from link where fromid=%d" % urlid).fetchone()[0] for i in range(iterations): print "Iteration %d" % i for urlid in pagerank: pr=0.15 for link in inlinks[urlid]: linkpr=pagerank[link] linkcount=numoutlinks[link] pr+=0.85*(linkpr/linkcount) pagerank[urlid]=pr for urlid in pagerank: self.con.execute("update pagerank set score=%f where urlid=%d" % (pagerank[urlid],urlid)) self.dbcommit() This function is 20 times faster (but uses a lot more memory for all the temporary dictionaries) because it avoids the unnecessary SQL queries in every iteration: >>> cProfile.run("crawler.calculatepagerank2()") 64802 function calls in 6.950 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.004 0.004 6.950 6.950 <string>:1(<module>) 1 1.004 1.004 6.946 6.946 searchengine.py:207(calculatepagerank2 2 0.000 0.000 0.104 0.052 searchengine.py:27(dbcommit) 23065 0.012 0.000 0.012 0.000 {meth 'append' of 'list' objects} 2 0.104 0.052 0.104 0.052 {meth 'commit' of 'sqlite3.Connection 1 0.000 0.000 0.000 0.000 {meth 'disable' of '_lsprof.Profiler' 31298 5.809 0.000 5.809 0.000 {meth 'execute' of 'sqlite3.Connectio 10431 0.018 0.000 0.018 0.000 {method 'fetchone' of 'sqlite3.Cursor' 1 0.000 0.000 0.000 0.000 {range} But is it possible to further reduce the number of SQL queries to speed up the function even more?

  Read the article

• Cygwin compiling error: "this application has requested the runtime to terminate it in an unusual wa

  - by naspinski

  I am trying to run Cygwin, and I am running into some problems. I tried to compile a program that works both on Windows with mingw and on a Unix system, but when I go to compile it through Cygwin: gcc threads.c -o threads I get the error: this application has requested the runtime to terminate it in an unusual way And that's it... any ideas what I am forgetting or screwing up? I haven't touched this stuff in about 5 years so I am beyond rusty; thanks!

  Read the article

• Building using CMake-GUI

  - by goodwince

  I am trying to build Marble on Windows (XP) and have been extremely unsuccessful. Following the instructions written here. I am wanting to build it as QT only. Configuration of the build tree fails. I am using the cmake-gui since I have never tried to compile using cmake it's possible it could be something in there. I found a similar issue; however, they compiled QT using Visual Studio and I am using QT Creator which compiles with MinGW so I need to compile in a different manner. CMake-GUI configuration source-code: MyDirectory/Marble where to build: MyDirectory/Build I also added an entry as.. DQTONLY and it's value set to ON

  Read the article

• Circular file references not allowed

  - by Program.X

  Hi, I am having a problem in building my solution in VS2008. Normally, it compiles fine in the environment. Sometimes, it fails with: /xxx_WEB/secure/CMSManagedTargetPage.aspx(1): error ASPPARSE: Circular file references are not allowed. I rebuild and it works fine. Now, however, I am in the middle of setting up a CruiseControl.NET system and am testing my checked out code with MSBuild before I integrate the build into CC. Now, everytime I MSBuild, I get: "Q:\cc\xxx\checked out from svn\xxx.sln" (default target) (1) -> (xxx_WEB target) -> /xxx_WEB/secure/CMSManagedTargetPage.aspx(1): error ASPPARSE: Circular file references are not allowed. Problem is, I can't see where this reference is. I have searched for the reference across the entire solution and canf ind no references to the page itself (CMSManagedTargetPage) anywhere other than in the page or its codebehind, or within a string, eg: C:\dev2008\xxx\IWW.xxx.ASPNET\AspxHttpHandler.cs(82): inputFile = context.Server.MapPath("~/secure/CMSManagedTargetPage.aspx"); C:\dev2008\xxx\IWW.xxx.ASPNET\AspxHttpHandler.cs(83): virtualPath = "~/secure/CMSManagedTargetPage.aspx"; My assembly references are also fine (as far as I know). My Web Application is at the "top" of the dependencies, and nothing references it and therefore the faulting page so cannot cause a circular reference. Of course, the page itself may reference something such as a UserControl within the same assembly/web site, but as mentioned earlier, a search on CMSManagedTargetPage yielded no results so this is not happening. Changing the batch attribute in web.config had no effect on MSBuild. I find it very odd that it "sometimes" fails in VS and always fails in MSBuild. Am I missing some subtlety?

  Read the article

• Prevent full table scan for query with multiple where clauses

  - by Dave Jarvis

  A while ago I posted a message about optimizing a query in MySQL. I have since ported the data and query to PostgreSQL, but now PostgreSQL has the same problem. The solution in MySQL was to force the optimizer to not optimize using STRAIGHT_JOIN. PostgreSQL offers no such option. Here is the explain: Here is the query: SELECT avg(d.amount) AS amount, y.year FROM station s, station_district sd, year_ref y, month_ref m, daily d LEFT JOIN city c ON c.id = 10663 WHERE -- Find all the stations within a specific unit radius ... -- 6371.009 * SQRT( POW(RADIANS(c.latitude_decimal - s.latitude_decimal), 2) + (COS(RADIANS(c.latitude_decimal + s.latitude_decimal) / 2) * POW(RADIANS(c.longitude_decimal - s.longitude_decimal), 2)) ) <= 50 AND -- Ignore stations outside the given elevations -- s.elevation BETWEEN 0 AND 2000 AND sd.id = s.station_district_id AND -- Gather all known years for that station ... -- y.station_district_id = sd.id AND -- The data before 1900 is shaky; insufficient after 2009. -- y.year BETWEEN 1980 AND 2000 AND -- Filtered by all known months ... -- m.year_ref_id = y.id AND m.month = 12 AND -- Whittled down by category ... -- m.category_id = '001' AND -- Into the valid daily climate data. -- m.id = d.month_ref_id AND d.daily_flag_id <> 'M' GROUP BY y.year It appears as though PostgreSQL is looking at the DAILY table first, which is simply not the right way to go about this query as there are nearly 300 million rows. How do I force PostgreSQL to start at the CITY table? Thank you!

  Read the article

< Previous Page | 71 72 73 74 75 76 77 78 79 80 81 82  | Next Page >