Search Results

Search found 20224 results on 809 pages for 'query optimization'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 181/809 | < Previous Page | 177 178 179 180 181 182 183 184 185 186 187 188  | Next Page >

  • optimize python code

    - by user283405
    i have code that uses BeautifulSoup library for parsing. But it is very slow. The code is written in such a way that threads cannot be used. Can anyone help me about this? I am using beautifulsoup library for parsing and than save in DB. if i comment the save statement, than still it takes time so there is no problem with database. def parse(self,text): soup = BeautifulSoup(text) arr = soup.findAll('tbody') for i in range(0,len(arr)-1): data=Data() soup2 = BeautifulSoup(str(arr[i])) arr2 = soup2.findAll('td') c=0 for j in arr2: if str(j).find("<a href=") > 0: data.sourceURL = self.getAttributeValue(str(j),'<a href="') else: if c == 2: data.Hits=j.renderContents() #and few others... #... c = c+1 data.save() Any suggestions? Note: I already ask this question here but that was closed due to incomplete information.

    Read the article

  • How to optimize a postgreSQL server for a "write once, read many"-type infrastructure ?

    - by mhu
    Greetings, I am working on a piece of software that logs entries (and related tagging) in a PostgreSQL database for storage and retrieval. We never update any data once it has been inserted; we might remove it when the entry gets too old, but this is done at most once a day. Stored entries can be retrieved by users. The insertion of new entries can happen rather fast and regularly, thus the database will commonly hold several millions elements. The tables used are pretty simple : one table for ids, raw content and insertion date; and one table storing tags and their values associated to an id. User search mostly concern tags values, so SELECTs usually consist of JOIN queries on ids on the two tables. To sum it up : 2 tables Lots of INSERT no UPDATE some DELETE, once a day at most some user-generated SELECT with JOIN huge data set What would an optimal server configuration (software and hardware, I assume for example that RAID10 could help) be for my PostgreSQL server, given these requirements ? By optimal, I mean one that allows SELECT queries taking a reasonably little amount of time. I can provide more information about the current setup (like tables, indexes ...) if needed.

    Read the article

  • Nginx , Apache , Mysql , Memcache with server 4G ram. How optimize to enoigh of memory?

    - by TomSawyer
    i have 1 dedicated server with Nginx proxy for Apache. Memcache, mysql, 4G Ram. These day, my visitor on my site wasn't increased, but my server get overload always in some specified time. (9AM - 15PM) Ram in use is increased second by second to full. that's moment, my server will get overload. i have to kill all apache , mysql service and reboot it to get free memory. and it'll full again. that's the terrible circle. here is my ram in use at the moment 160(nginx) 220(apache) 512(memcache) 924(mysql) here's process number 4(nginx) 14(apache) 5(memcache) 20(mysql) and here's my my.cnf config. someone can help me to optimize it? [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql skip-locking skip-networking skip-name-resolve # enable log-slow-queries log-slow-queries = /var/log/mysql-slow-queries.log long_query_time=3 max_connections=200 wait_timeout=64 connect_timeout = 10 interactive_timeout = 25 thread_stack = 512K max_allowed_packet=16M table_cache=1500 read_buffer_size=4M join_buffer_size=4M sort_buffer_size=4M read_rnd_buffer_size = 4M max_heap_table_size=256M tmp_table_size=256M thread_cache=256 query_cache_type=1 query_cache_limit=4M query_cache_size=16M thread_concurrency=8 myisam_sort_buffer_size=128M # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 [mysqldump] quick max_allowed_packet=16M [mysql] no-auto-rehash [isamchk] key_buffer=256M sort_buffer=256M read_buffer=64M write_buffer=64M [myisamchk] key_buffer=256M sort_buffer=256M read_buffer=64M write_buffer=64M [mysqlhotcopy] interactive-timeout [mysql.server] user=mysql basedir=/var/lib [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

    Read the article

  • Limit CPU usage of a process

    - by jb
    I have a service running which periodically checks a folder for a file and then processes it. (Reads it, extracts the data, stores it in sql) So I ran it on a test box and it took a little longer thaan expected. The file had 1.6 million rows, and it was still running after 6 hours (then I went home). The problem is the box it is running on is now absolutely crippled - remote desktop was timing out so I cant even get on it to stop the process, or attach a debugger to see how far through etc. It's solidly using 90%+ CPU, and all other running services or apps are suffering. The code is (from memory, may not compile): List<ItemDTO> items = new List<ItemDTO>(); using (StreamReader sr = fileInfo.OpenText()) { while (!sr.EndOfFile) { string line = sr.ReadLine() try { string s = line.Substring(0,8); double y = Double.Parse(line.Substring(8,7)); //If the item isnt already in the collection, add it. if (items.Find(delegate(ItemDTO i) { return (i.Item == s); }) == null) items.Add(new ItemDTO(s,y)); } catch { /*Crash*/ } } return items; } - So I am working on improving the code (any tips appreciated). But it still could be a slow affair, which is fine, I've no problems with it taking a long time as long as its not killing my server. So what I want from you fine people is: 1) Is my code hideously un-optimized? 2) Can I limit the amount of CPU my code block may use? Cheers all

    Read the article

  • Where does the compiler store methods for C++ classes?

    - by Mashmagar
    This is more a curiosity than anything else... Suppose I have a C++ class Kitty as follows: class Kitty { void Meow() { //Do stuff } } Does the compiler place the code for Meow() in every instance of Kitty? Obviously repeating the same code everywhere requires more memory. But on the other hand, branching to a relative location in nearby memory requires fewer assembly instructions than branching to an absolute location in memory on modern processors, so this is potentially faster. I suppose this is an implementation detail, so different compilers may perform differently. Keep in mind, I'm not considering static or virtual methods here.

    Read the article

  • Optimize Binary Search Algorithm

    - by Ganesh M
    In a binary search, we have two comparisons one for greater than and other for less than, otherwise its the mid value. How would you optimize so that we need to check only once? bool binSearch(int array[], int key, int left, int right) { mid = left + (right-left)/2; if (key < array[mid]) return binSearch(array, key, left, mid-1); else if (key > array[mid]) return binSearch(array, key, mid+1, right); else if (key == array[mid]) return TRUE; // Found return FALSE; // Not Found }

    Read the article

  • Is the C++ compiler optimizer allowed to break my destructor ability to be called multiple times?

    - by sharptooth
    We once had an interview with a very experienced C++ developer who couldn't answer the following question: is it necessary to call the base class destructor from the derived class destructor in C++? Obviously the answer is no, C++ will call the base class destructor automagically anyway. But what if we attempt to do the call? As I see it the result will depend on whether the base class destructor can be called twice without invoking erroneous behavior. For example in this case: class BaseSafe { public: ~BaseSafe() { } private: int data; }; class DerivedSafe { public: ~DerivedSafe() { BaseSafe::~BaseSafe(); } }; everything will be fine - the BaseSafe destructor can be called twice safely and the program will run allright. But in this case: class BaseUnsafe { public: BaseUnsafe() { buffer = new char[100]; } ~BaseUnsafe () { delete[] buffer; } private: char* buffer; }; class DerivedUnsafe { public: ~DerivedUnsafe () { BaseUnsafe::~BaseUnsafe(); } }; the explicic call will run fine, but then the implicit (automagic) call to the destructor will trigger double-delete and undefined behavior. Looks like it is easy to avoid the UB in the second case. Just set buffer to null pointer after delete[]. But will this help? I mean the destructor is expected to only be run once on a fully constructed object, so the optimizer could decide that setting buffer to null pointer makes no sense and eliminate that code exposing the program to double-delete. Is the compiler allowed to do that?

    Read the article

  • SQL query to select distinct record with 2 or more repetition in another field

    - by kyohiros
    So I have this table of book orders, it contains 2 columns, one is the order ID(primary key) and another is the ID of the book that the customer ordered. For example: | OrderID | BookID | | 0001 | B002 | | 0002 | B005 | | 0003 | B002 | | 0004 | B003 | | 0005 | B005 | | 0006 | B002 | | 0007 | B002 | What I want is to get the IDs of the books that got 2 or more purchases/orders, for example if I run the SQL query against the above data, I would get this as the result: | BookID | | B002 | | B005 | I don't know if this can be archived in SQL or I have to built a simpler statement and repetitive run the query against all the records in another language. I need some help and thanks for reading my question.

    Read the article

  • STL vectors with uninitialized storage?

    - by Jim Hunziker
    I'm writing an inner loop that needs to place structs in contiguous storage. I don't know how many of these structs there will be ahead of time. My problem is that STL's vector initializes its values to 0, so no matter what I do, I incur the cost of the initialization plus the cost of setting the struct's members to their values. Is there any way to prevent the initialization, or is there an STL-like container out there with resizeable contiguous storage and uninitialized elements? (I'm certain that this part of the code needs to be optimized, and I'm certain that the initialization is a significant cost.) Also, see my comments below for a clarification about when the initialization occurs. SOME CODE: void GetsCalledALot(int* data1, int* data2, int count) { int mvSize = memberVector.size() memberVector.resize(mvSize + count); // causes 0-initialization for (int i = 0; i < count; ++i) { memberVector[mvSize + i].d1 = data1[i]; memberVector[mvSize + i].d2 = data2[i]; } }

    Read the article

  • Referencing outer query's tables in a subquery

    - by soulmerge
    Is it possible to reference an outer query in a subquery with MySQL? I know there are some cases where this is possible: SELECT * FROM table t1 WHERE t1.date = ( SELECT MAX(date) FROM table t2 WHERE t2.id = t1.id)` ); But I'm wondering if something like this could work: SELECT u.username, c._postCount FROM User u INNER JOIN ( SELECT p.user, COUNT(*) AS _postCount FROM Posting p --# This is the reference I would need: WHERE p.user = u.id ) c ON c.user = u.id WHERE u.joinDate < '2009-10-10'; I know I could achieve the same using a GROUP BY or by pulling the outer WHERE clause into the sub-query, but I need this for automatic SQL generation and cannot use either alternative for various other reasons.

    Read the article

  • Java: how to read BufferedReader faster

    - by Cata
    Hello, I want to optimize this code: InputStream is = rp.getEntity().getContent(); BufferedReader reader = new BufferedReader(new InputStreamReader(is)); String text = ""; String aux = ""; while ((aux = reader.readLine()) != null) { text += aux; } The thing is that i don't know how to read the content of the bufferedreader and copy it in a String faster than what I have above. I need to spend as little time as possible. Thank you

    Read the article

  • How can I improve the below query?

    - by Newbie
    I have the following input. INPUT: TableA ID Sentences --- ---------- 1 I am a student 2 Have a nice time guys! What I need to do is to extract the words from the sentence(s) and insert each individual word in another table OUTPUT: SentenceID WordOccurance Word ---------- ------------ ----- 1 1 I 1 2 am 1 3 a 1 4 student 2 1 Have 2 2 a 2 3 nice 2 4 time 2 5 guys! I was able to get the answer by using the below query ;With numCTE As ( Select rn = 1 Union all Select rn+1 from numCTE where rn<1000) select SentenceID=id, WordOccurance=row_number()over(partition by TableA.ID order by rn), Word = substring(' '+sentences+' ', rn+1, charindex(' ',' '+sentences+' ', rn+1)-rn-1) from TableA join numCTE on rn <= len(' '+sentences+' ') where substring(' '+sentences+' ', rn,1) = ' ' order by id, rn How can I improve this query of mine.? Basically I am looking for a better solution than the one presented Thanks

    Read the article

  • I'm doing a lot of lists and dictionary sorting...and this is causing memory errors in Python websit

    - by alex
    I retrieved data from the log table in my database. Then I started finding unique users, comparing/sorting lists, etc. In the end I got down to this. stats = {'2010-03-19': {'date': '2010-03-19', 'unique_users': 312, 'queries': 1465}, '2010-03-18': {'date': '2010-03-18', 'unique_users': 329, 'queries': 1659}, '2010-03-17': {'date': '2010-03-17', 'unique_users': 379, 'queries': 1845}, '2010-03-16': {'date': '2010-03-16', 'unique_users': 434, 'queries': 2336}, '2010-03-15': {'date': '2010-03-15', 'unique_users': 390, 'queries': 2138}, '2010-03-14': {'date': '2010-03-14', 'unique_users': 460, 'queries': 2221}, '2010-03-13': {'date': '2010-03-13', 'unique_users': 507, 'queries': 2242}, '2010-03-12': {'date': '2010-03-12', 'unique_users': 629, 'queries': 3523}, '2010-03-11': {'date': '2010-03-11', 'unique_users': 811, 'queries': 4274}, '2010-03-10': {'date': '2010-03-10', 'unique_users': 171, 'queries': 1297}, '2010-03-26': {'date': '2010-03-26', 'unique_users': 299, 'queries': 1617}, '2010-03-27': {'date': '2010-03-27', 'unique_users': 323, 'queries': 1310}, '2010-03-24': {'date': '2010-03-24', 'unique_users': 352, 'queries': 2112}, '2010-03-25': {'date': '2010-03-25', 'unique_users': 330, 'queries': 1290}, '2010-03-22': {'date': '2010-03-22', 'unique_users': 329, 'queries': 1798}, '2010-03-23': {'date': '2010-03-23', 'unique_users': 329, 'queries': 1857}, '2010-03-20': {'date': '2010-03-20', 'unique_users': 368, 'queries': 1693}, '2010-03-21': {'date': '2010-03-21', 'unique_users': 329, 'queries': 1511}, '2010-03-29': {'date': '2010-03-29', 'unique_users': 325, 'queries': 1718}, '2010-03-28': {'date': '2010-03-28', 'unique_users': 340, 'queries': 1815}, '2010-03-30': {'date': '2010-03-30', 'unique_users': 329, 'queries': 1891}} It's not a big dictionary. But when I try to do one last thing...it craps out on me. for k, v in stats: mylist.append(v) too many values to unpack What the heck does that mean??? TOO MANY VALUES TO UNPACK.

    Read the article

  • PHP what is faster to use

    - by user1631500
    What is faster / better to use? To put html into variables and print them later, or to just html print / echo print the content based on condition? EXAMPLE 1:(html into variables) if(!isset($_SESSION['daddy'])) { $var = "<span class='something'>Go here:<a href='#'>Click</a></span> <span class='something'>Go here:<a href='#'>Click</a></span>" } else { $one=$_SESSION["one"]; $two=$_SESSION["two"]; $three=$_SESSION["three"]; $var = You are cool enough to view the content; } echo $var; EXAMPLE 2:(print based on condition) if(!isset($_SESSION['daddy'])) { $var = 1; } else { $one=$_SESSION["one"]; $two=$_SESSION["two"]; $three=$_SESSION["three"]; $var = 0; } if ($var==1) { ?> <span class='something'>Go here:<a href='#'>Click</a></span> <span class='something'>Go here:<a href='#'>Click</a></span <?php } else { ?> You are cool enough to view the content. <?php } ?>

    Read the article

  • .net Compiler Optimizations

    - by Dested
    I am writing an application that I need to run at incredibly low speeds. The application creates and destroys memory in creative ways throughout its run, and it works just fine. I am wondering what compiler optimizations occur so I can try to build to that. One trick off hand is that the CLR handles arrays much faster than lists, so if you need to handle a ton of elements in a List, you may be better off calling ToArray() and handling it rather than calling ElementAt() again and again. I am wondering if there is any sort of comprehensive list for this kind of thing, or maybe the SO community can create one :-)

    Read the article

  • PHP error handling : my code is not optimized

    - by Tristan
    Hello, I must warn you, this code will heart your eyes, so please don't judge me, i'm trying to improve the way I handle errors all my tests are like this : if ($something < 27) { $error_IP= '<div class="error_message">something bad</div> '; }else{ $erreur_IP=''; } and here's the ugliest thing : if( !isset($_POST) || ($erreur_captcha !='') || ($erreur_email !='') || ($erreur_hebergeurVide != '') || ($erreur_paysVide != '') || ($erreur_slotVide != '') || ($erreur_rconVide != '') || ($erreur_tick != '') + a lot more :d ) What do you suggest to me to optimize my errors handling ? Thank you

    Read the article

  • Use SQL query to populate property in nHibernate mapping file

    - by brainimus
    I have an object which contains a property that is the result of an SQL statement. How do I add the SQL statement to my nHibernate mapping file? Example Object: public class Library{ public int BookCount { get; set; } } Example Mapping File: <hibernate-mapping> <class name="Library" table="Libraries"> <property name="BookCount" type="int"> <- This is where I want the SQL query to populate the value. -> </class> </hibernate-mapping> Example SQL Query: SELECT COUNT(*) FROM BOOKS WHERE BOOKS.LIBRARY_ID = LIBRARIES.ID

    Read the article

  • Fastest way to find the rotation of a vector

    - by kriss
    I have two 2D vectors, say u and v, defined by cartesian coordinates. Imagine that vectors are needles of a clock. I'm looking for the fastest way to find out, using python, if v is after or before u (or in other words find out in wich half plane is v, regarding to position of u). For the purpose of the problem if vectors are aligned answer should be before. It seems easy using some trigonometry, but I believe there should be a faster way using coordinates only. My test case: def after(u, v): """code here""" after((4,2), (6, 1)) : True after((4,2), (3, 3)) : False after((4,2), (2, 1)) : False after((4,2), (3, -3)) : True after((4,2), (-2, -5)) : True after((4,2), (-4, -2)) : False

    Read the article

  • 50 million+ Rows of Data - CSV or MySQL

    - by eWizardII
    Hello, I have a CSV file which is about 1GB big and contains about 50million rows of data, I am wondering is it better to keep it as a CSV file or store it as some form of a database. I don't know a great deal about MySQL to argue for why I should use it or another database framework over just keeping it as a CSV file. I am basically doing a Breadth-First Search with this dataset, so once I get the initial "seed" set the 50million I use this as the first values in my queue. Thanks,

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 177 178 179 180 181 182 183 184 185 186 187 188  | Next Page >