Search Results

Search found 28685 results on 1148 pages for 'query performance'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 238/1148 | < Previous Page | 234 235 236 237 238 239 240 241 242 243 244 245  | Next Page >

  • Improving performance for WRITE operation on Oracle DB in Java

    - by Lucky
    I've a typical scenario & need to understand best possible way to handle this, so here it goes - I'm developing a solution that will retrieve data from a remote SOAP based web service & will then push this data to an Oracle database on network. Also, this will be a scheduled task that will execute every 15 minutes. I've event queues on remote service that contains the INSERT/UPDATE/DELETE operations that have been done since last retrieval, & once I retrieve the events for last 15 minutes, it again add events for next retrieval. Now, its just pushing data to Oracle so all my interactions are INSERT & UPDATE statements. There are around 60 tables on Oracle with some of them having 100+ columns. Moreover, for every 15 minutes cycle there would be around 60-70 Inserts, 100+ Updates & 10-20 Deletes. This will be an executable jar file that will terminate after operation & will again start on next 15 minutes cycle. So, I need to understand how should I handle WRITE operations (best practices) to improve performance for this application as whole ? Current Test Code (on every cycle) - Connects to remote service to get events. Creates a connection with DB (single connection object). Identifies the type of operation (INSERT/UPDATE/DELETE) & table on which it is done. After above, calls the respective method based on type of operation & table. Uses Preparedstatement with positional parameters, & retrieves each column value from remote service & assigns that to statement parameters. Commits the statement & returns to get event class to process next event. Above is repeated till all the retrieved events are processed after which program closes & then starts on next cycle & everything repeats again. Thanks for help !

    Read the article

  • Best way to store data in database when you don't know the type

    - by stiank81
    I have a table in my database that represents datafields in a custom form. The DataField gives some representation of what kind of control it should be represented with, and what value type it should take. Simplified you can say that I have 2 entities in this table - Textbox taking any string and Textbox only taking numbers. Now I have the different values stored in a separate table, referencing the datafield definition. What is the best way to store the data value here, when the type differs? One possible solution is to have the FieldValue table hold one field per possible value type. Now this would certainly be redundant, but at least I would get the value stored in its correct form - simplifying queries later. FieldValue ---------- Id DataFieldId IntValue DoubleValue BoolValue DataValue .. Another possibility is just storing everything as String, and casting this in the queries. I am using .Net with NHibernate, and I see that at least here there is a Projections.Cast that can be used to cast e.g. string to int in the query. Either way in these two solutions I need to know which type to use when doing the query, but I will know that from the DataField, so that won't be a problem. Anyway; I don't think any of these solutions sounds good. Are they? Or is there a better way?

    Read the article

  • Adding up row number and displaying total using COUNT (PHP MySQL)

    - by Yvonne
    I'm attempting to run a query that adds up the total number of subjects in a class. A class has many subjects. There is a 'teachersclasses' table between teachers (the user table) and classes. The principles sounds pretty simple but I'm having some trouble in getting my page to display the number of subjects for each class (directly associated with the teacher) This is what I have so far, trying to make use of the COUNT with a nested SELECT: SELECT (SELECT count(*) FROM subjects WHERE subjects.classid = class.classid) AS total_subjects, class.classname, class.classid FROM class Then I am calling up 'num_subjects' to present the total within a while loop: <?php echo $row['total_subjects']?> From the above, I am receiving the total subjects for a class, but within the same table row (for one class) and my other while loop doesnt run anymore, which returns all of the classes associated with a teacher :( ... Bit of a mess now! I know to return the classes for a particular teacher, I can do an additional WHERE clause on the session of 'teacherid' but I think my query is getting too complicated for me that errors are popping up everywhere. Anyone have a quick fix for this! Thanks very much

    Read the article

  • SDK2 query for counting: which is more efficient?

    - by user1195996
    I have an app that is displaying metrics about defects in a project. I have the option of making one query that returns all the defects, and from that I can break out about four different metrics (How many defects escaped QA in 90 days, 180 days, and then the same metrics again but only counting sev1/sev2 defects). I could make four queries and limit the results to one so that I just get a count for each. Or I could make one query that encompass them all (all defects that escaped QA in 180 days) and then count up the difference. I'm figuring worst case, the number of defects that escaped QA in the last six months will generally be less than 100, certainly less 500 worst case. Which would you do-- four queryies with one result each, or one single query that on average might return 50, perhaps worst case 500? And I guess the key question is-- where are the inflections points? Perhaps I have more metrics tomorrow (who knows, 8?) and a different average defect counts. Is there a rule of thumb I could use to help choose which approach?

    Read the article

  • MySQLi - Should every statement be prepared?

    - by Kerry
    I know its supposed to improve performance and clean strings, but lets say there are no variables? Might just be a SELECT COUNT( `column` ) AS count FROM `table` Should that be prepared? Is there any case that a SELECT statement should not be prepared?

    Read the article

  • How much slower is a try/catch block? [closed]

    - by Euclid
    Possible Duplicate: What is the real overhead of try/catch in C#? how much slower is a try catch block than a conditional? eg try { v = someArray[10]; } catch { v = defaultValue; } or if (null != someArray) { v = someArray[10]; } else { v = defaultValue; } is there much in it or isn't there a definative performance differance?

    Read the article

  • Query total page count via SNMP HP Laserjet

    - by Tim
    I was asked to get hold of the total pages counts for the 100+ printers we have at work. All of them are HP Laser or Business Jets of some description and the vast majority are connected via some form of HP JetDirect network card/switch. After many hours of typing in IP addresses and copying and pasting the relevant figure in to Excel I have now been asked to do this on a weekly basis. This led me to think there must be an easier way, as an IT professional I can surely work out some time saving method to solve this issue. Suffice it to say I do not feel very professional now after a day or so of trying to make SNMP work for me! From what I understand the first thing is to enable SNMP on the printer. Done. Next I would need something to query the SNMP bit. I decided to go open source and free and someone here recommended net-snmp as a decent tool (I would like to have just added the printers as nodes in SolarWinds but we are somewhat tight on licences apparently). Next I need the name of the MIB. For this I believe the HP-LASERJET-COMMON-MIB has the correct information in it. Downloaded this and added to net-snmp. Now I need the OID which I believe after much scouring is printed-media-simplex-count (we have no duplex printers, that we are interested in at least). Running the following command yields the following demoralising output: snmpget -v 2c -c public 10.168.5.1 HP-LASERJET-COMMON-MIB:.1.3.6.1.2.1.1.16.1.1.1 (the OID was derived from running: snmptranslate -IR -On printed-media-simplex-count Unlinked OID in HP-LASERJET-COMMON-MIB: hp ::= { enterprises 11 } Undefined identifier: enterprises near line 3 of C:/usr/share/snmp/mibs/HP-LASER JET-COMMON-MIB..txt .1.3.6.1.2.1.1.16.1.1.1 ) Unlinked OID in HP-LASERJET-COMMON-MIB: hp ::= { enterprises 11 } Undefined identifier: enterprises near line 3 of C:/usr/share/snmp/mibs/HP-LASER JET-COMMON-MIB..txt HP-LASERJET-COMMON-MIB:.1.3.6.1.2.1.1.16.1.1.1: Am I barking up the wrong tree completely with this? My aim was to script it all to output to a file for all the IP addresses of the printers and then plonk that in Excel for my lords and masters to digest at their leisure. I have a feeling I am using either the wrong MIB or the wrong OID from said MIB (or both). Does anyone have any pointers on this for me? Or should I give up and go back to navigationg each printers web page individually (hoping not).

    Read the article

  • nginx rewrite rule to convert URL segments to query string parameters

    - by Nick
    I'm setting up an nginx server for the first time, and having some trouble getting the rewrite rules right for nginx. The Apache rules we used were: See if it's a real file or directory, if so, serve it, then send all requests for / to Director.php DirectoryIndex Director.php If the URL has one segment, pass it as rt RewriteRule ^/([a-zA-Z0-9\-\_]+)/$ /Director.php?rt=$1 [L,QSA] If the URL has two segments, pass it as rt and action RewriteRule ^/([a-zA-Z0-9\-\_]+)/([a-zA-Z0-9\-\_]+)/$ /Director.php?rt=$1&action=$2 [L,QSA] My nginx config file looks like: server { ... location / { try_files $uri $uri/ /index.php; } location ~ \.php$ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; } } How do I get the URL segments into Query String Parameters like in the Apache rules above? UPDATE 1 Trying Pothi's approach: # serve static files directly location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|html)$ { access_log off; expires 30d; } location / { try_files $uri $uri/ /Director.php; rewrite "^/([a-zA-Z0-9\-\_]+)/$" "/Director.php?rt=$1" last; rewrite "^/([a-zA-Z0-9\-\_]+)/([a-zA-Z0-9\-\_]+)/$" "/Director.php?rt=$1&action=$2" last; } location ~ \.php$ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; } This produces the output No input file specified. on every request. I'm not clear on if the .php location gets triggered (and subsequently passed to php) when a rewrite in any block indicates a .php file or not. UPDATE 2 I'm still confused on how to setup these location blocks and pass the parameters. location /([a-zA-Z0-9\-\_]+)/ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME ${document_root}Director.php?rt=$1{$args}; include fastcgi_params; } UPDATE 3 It looks like the root directive was missing, which caused the No input file specified. message. Now that this is fixed, I get the index file as if the URL were / on every request regardless of the number of URL segments. It appears that my location regular expression is being ignored. My current config is: # This location is ignored: location /([a-zA-Z0-9\-\_]+)/ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index Director.php; set $args $query_string&rt=$1; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; } location / { try_files $uri $uri/ /Director.php; } location ~ \.php$ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index Director.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; }

    Read the article

  • Access Qry Questions

    - by kralco626
    It was suggested that I repost this questions as I didn't do a very good job discribing my issue the first time. (http://stackoverflow.com/questions/2921286/access-question) THE SITUATION: I have inspections from many months of many years. Sometimes there is more than one inspection in a month, sometimes there is no inspection. However, the report that is desired by the clients requires that I have EXACTLY ONE record per month for the time frame they request the report. They understand the data issues and have stated that if there is more than one inspection in a month to take the latest one. If the is not an inspection for that month, go back in time untill you find one and use that one. So a sample of the data is as follows: (I am including many records because I was told I did not include enough data on my last try) equip_id month year runtime date 1 5 2008 400 5/10/2008 12:34 PM 1 7 2008 500 7/12/2008 1:45 PM 1 8 2008 600 8/20/2008 1:12 PM 1 8 2008 605 8/30/2008 8:00 AM 1 1 2010 2000 1/12/2010 2:00 PM 1 3 2010 2200 3/24/2010 10:00 AM 2 7 2009 1000 7/20/2009 8:00 AM 2 10 2009 1400 10/14/2009 9:00 AM 2 1 2010 1600 1/15/2010 1:00 PM 2 1 2010 1610 1/30/2010 4:00 PM 2 3 2010 1800 3/15/2010 1:00PM After all the transformations to the data are done, it should look like this: equip_id month year runtime date 1 5 2008 400 5/10/2008 12:34 PM 1 6 2008 400 5/10/2008 12:34 PM 1 7 2008 500 7/12/2008 1:45 PM 1 8 2008 605 8/30/2008 8:00 AM 1 9 2008 605 8/30/2008 8:00 AM 1 10 2008 605 8/30/2008 8:00 AM 1 11 2008 605 8/30/2008 8:00 AM 1 12 2008 605 8/30/2008 8:00 AM 1 1 2009 605 8/30/2008 8:00 AM 1 2 2009 605 8/30/2008 8:00 AM 1 3 2009 605 8/30/2008 8:00 AM 1 4 2009 605 8/30/2008 8:00 AM 1 5 2009 605 8/30/2008 8:00 AM 1 6 2009 605 8/30/2008 8:00 AM 1 7 2009 605 8/30/2008 8:00 AM 1 8 2009 605 8/30/2008 8:00 AM 1 9 2009 605 8/30/2008 8:00 AM 1 10 2009 605 8/30/2008 8:00 AM 1 11 2009 605 8/30/2008 8:00 AM 1 12 2009 605 8/30/2008 8:00 AM 1 1 2010 2000 1/12/2010 2:00 PM 1 2 2010 2000 1/12/2010 2:00 PM 1 3 2010 2200 3/24/2010 10:00 AM 2 7 2009 1000 7/20/2009 8:00 AM 2 8 2009 1000 7/20/2009 8:00 AM 2 9 2009 1000 7/20/2009 8:00 AM 2 10 2009 1400 10/14/2009 9:00 AM 2 11 2009 1400 10/14/2009 9:00 AM 2 12 2009 1400 10/14/2009 9:00 AM 2 1 2010 1610 1/30/2010 4:00 PM 2 2 2010 1610 1/30/2010 4:00 PM 2 3 2010 1800 3/15/2010 1:00PM I think that this is the most accurate dipiction of the problem that I can give. I will now say what I have tried. Although if someone else has a better approach, I am perfectly willing to throw away what I have done and do it differently... STEP 1: create a query that removes the duplicates from the data. Ie. only one record per equip_id for each month/year, keeping the latest one. (done successfully) STEP 2: create a table of the date ranges the client wants the report for. (This is done dynamically at runtime) This table two field, Month and Year. So if the client wants a report from FEb 2008 to March 2010 the table would look like: Month Year 2 2008 3 2008 . . . 12 2008 1 2009 . . . 12 2009 1 2010 2 2010 3 2010 I then left joined this table with my query from step 1. So now I have a record for every month and every year that they want the report for, with nulls(or blanks) or sometimes 0s (not sure why, access is weird, but sometiems they are nulls and sumtimes they are 0s...) for the runtimes that are not avaiable. I don't particurally like this solution, but ill do it if i have to. (this is also done successfully) STEP 3: Fill in the missing runtime values. This I HAVE NOT done successfully. Note that if the request range for the report is feb 2008 to march 2010 and the oldest record for a particular equip_id is say june 2008, it is O.K. for the runtimes to be null (or zeros) for feb - may 2008. I am working with the following query for this step: SELECT equip_id as e_id,year,month, (select top 1 runhours from qry_1_c_One_Record_per_Month a where a.equip_id = e_id order by year,month) FROM qry_1_c_One_Record_per_Month where runhours is null or runhours = 0; UNION SELECT equip_id, year, month, runhours FROM qry_1_c_One_Record_per_Month WHERE .runhours Is Not Null And runhours <> 0 However I clearly can't check the a.equip_id = e_id ... so i don't have anyway to make sure i'm looking at the correct equip_id SUMMARY: So like i said i'm willing to throw away any part, or all of what I tried. Just trying to give everyone a complete picture. I REALLY apreciate ANY help! Thanks so much in advance!

    Read the article

  • LDAP query on linux against AD returns groups with no members

    - by SethG
    I am using LDAP+kerberos to authenticate against Active Directory on Windows 2003 R2. My krb5.conf and ldap.conf appear to be correct (according to pretty much every sample I found on the 'net). I can login to the host with both password and ssh keys. When I run getent passwd, all my ldap user accounts are listed with all the important attributes. When I run getent group, all the ldap groups and their gid's are listed, but no group members. If I run ldapsearch and filter on any group, the members are all listed with the "member" attribute. So the data is there for the taking, it's just not being parsed properly. It would appear that I simply am using an incorrect mapping in ldap.conf, but I can't see it. I've tried several variations and all give the same result. Here is my current ldap.conf: host <ad-host1-ip> <ad-host2-ip> base dc=my,dc=full,dc=dn uri ldap://<ad-host1> ldap://<ad-host2> ldap_version 3 binddn <mybinddn> bindpw <mybindpw> scope sub bind_policy hard nss_reconnect_tries 3 nss_reconnect_sleeptime 1 nss_reconnect_maxsleeptime 8 nss_reconnect_maxconntries 3 nss_map_objectclass posixAccount User nss_map_objectclass posixGroup Group nss_map_attribute uid sAMAccountName nss_map_attribute gidNumber msSFU30GidNumber nss_map_attribute uidNumber msSFU30UidNumber nss_map_attribute cn cn nss_map_attribute gecos displayName nss_map_attribute homeDirectory msSFU30HomeDirectory nss_map_attribute loginShell msSFU30LoginShell nss_map_attribute uniqueMember member pam_filter objectcategory=User pam_login_attribute sAMAccountName pam_member_attribute member pam_password ad Here's the kicker: this config works 100% fine on a different linux box with a different distro. It does not work on the distro I am planning on switching to. I have installed from source the versions of pam_ldap and nss_ldap on the new box to match the old box, which fixed another problem I was having with this setup. Other relevant info is the original AD box was Windows 2003. It's mirror died a horrible hardware death so I'm trying to add two more 2003-R2 servers to the mirror tree and ultimately drop the old 2003 box. The new R2 boxes appear to have joined the DC forest properly. What do I need to do to get groups working? I've exhausted all the resources I could find and need a different angle. Any input is appreciated. Status update, 7/31/09 I have managed to tweak my config file to get full info from the AD and performance is nice and snappy. I replaced the back-rev'd copies of pam_ldap and nss_ldap with the current ones for the distro I'm using, so it's back to a standard out-of-the-box install. Here's my current config: host <ad-host1-ip> <ad-host2-ip> base dc=my,dc=full,dc=dn uri ldap://<ad-host1> ldap://<ad-host2> ldap_version 3 binddn <mybinddn> bindpw <mybindpw> scope sub bind_policy soft nss_reconnect_tries 3 nss_reconnect_sleeptime 1 nss_reconnect_maxsleeptime 8 nss_reconnect_maxconntries 3 nss_connect_policy oneshot referrals no nss_map_objectclass posixAccount User nss_map_objectclass posixGroup Group nss_map_attribute uid sAMAccountName nss_map_attribute gidNumber msSFU30GidNumber nss_map_attribute uidNumber msSFU30UidNumber nss_map_attribute cn cn nss_map_attribute gecos displayName nss_map_attribute homeDirectory msSFU30HomeDirectory nss_map_attribute loginShell msSFU30LoginShell nss_map_attribute uniqueMember member pam_filter objectcategory=CN=Person,CN=Schema,CN=Configuration,DC=w2k,DC=cis,DC=ksu,DC=edu pam_login_attribute sAMAccountName pam_member_attribute member pam_password ad ssl off tls_checkpeer no sasl_secprops maxssf=0 The remaining problem now is when you run the groups command, not all subscribed groups are listed. Some are (one or two), but not all. Group memberships are still honored, such as file and printer access. getent group foo still shows that the user is a member of group foo. So it appears to be a presentation bug, and does not interfere with normal operation. It also appears that some (I have not determined exactly how many) group searches do not resolve correctly, even though the group is listed. eg, when you run "getent group bar", nothing is returned, but if you run "getent group|grep bar" or "getent group|grep <bar_gid>" you can see that it indeed listed and your group name and gid are correct. This still seems like an LDAP search or mapping error, but I can't figure out what it is. I'm a heckuva lot closer than earlier in the week, but I'd really like to get this last detail ironed out.

    Read the article

  • Regular expression to remove one parameter from query string

    - by Kip
    I'm looking for a regular expression to remove a single parameter from a query string, and I want to do it in a single regular expression if possible. Say I want to remove the foo parameter. Right now I use this: /&?foo\=[^&]+/ That works as long as foo is not the first parameter in the query string. If it is, then my new query string starts with an ampersand. (For example, "foo=123&bar=456" gives a result of "&bar=456".) Right now, I'm just checking after the regex if the query string starts with ampersand, and chopping it off if it does. Example edge cases: Input | Output -------------------------+----------------- foo=123 | (empty string) foo=123&bar=456 | bar=456 bar=456&foo=123 | bar=456 abc=789&foo=123&bar=456 | abc=789&bar=456 Edit OK as pointed out in comments there are there are way more edge cases than I originally considered. I got the following regex to work with all of them: /&foo(\=[^&]*)?(?=&|$)|^foo(\=[^&]*)?(&|$)/ This is modified from Mark Byers's answer, which is why I'm accepting that one, but Roger Pate's input helped a lot too. Here is the full suite of test cases I'm using, and a Perl script which tests them. Input | Output -------------------------+------------------- foo | foo&bar=456 | bar=456 bar=456&foo | bar=456 abc=789&foo&bar=456 | abc=789&bar=456 foo= | foo=&bar=456 | bar=456 bar=456&foo= | bar=456 abc=789&foo=&bar=456 | abc=789&bar=456 foo=123 | foo=123&bar=456 | bar=456 bar=456&foo=123 | bar=456 abc=789&foo=123&bar=456 | abc=789&bar=456 xfoo | xfoo xfoo&bar=456 | xfoo&bar=456 bar=456&xfoo | bar=456&xfoo abc=789&xfoo&bar=456 | abc=789&xfoo&bar=456 xfoo= | xfoo= xfoo=&bar=456 | xfoo=&bar=456 bar=456&xfoo= | bar=456&xfoo= abc=789&xfoo=&bar=456 | abc=789&xfoo=&bar=456 xfoo=123 | xfoo=123 xfoo=123&bar=456 | xfoo=123&bar=456 bar=456&xfoo=123 | bar=456&xfoo=123 abc=789&xfoo=123&bar=456 | abc=789&xfoo=123&bar=456 foox | foox foox&bar=456 | foox&bar=456 bar=456&foox | bar=456&foox abc=789&foox&bar=456 | abc=789&foox&bar=456 foox= | foox= foox=&bar=456 | foox=&bar=456 bar=456&foox= | bar=456&foox= abc=789&foox=&bar=456 | abc=789&foox=&bar=456 foox=123 | foox=123 foox=123&bar=456 | foox=123&bar=456 bar=456&foox=123 | bar=456&foox=123 abc=789&foox=123&bar=456 | abc=789&foox=123&bar=456 Test script (Perl) @in = ('foo' , 'foo&bar=456' , 'bar=456&foo' , 'abc=789&foo&bar=456' ,'foo=' , 'foo=&bar=456' , 'bar=456&foo=' , 'abc=789&foo=&bar=456' ,'foo=123' , 'foo=123&bar=456' , 'bar=456&foo=123' , 'abc=789&foo=123&bar=456' ,'xfoo' , 'xfoo&bar=456' , 'bar=456&xfoo' , 'abc=789&xfoo&bar=456' ,'xfoo=' , 'xfoo=&bar=456' , 'bar=456&xfoo=' , 'abc=789&xfoo=&bar=456' ,'xfoo=123', 'xfoo=123&bar=456', 'bar=456&xfoo=123', 'abc=789&xfoo=123&bar=456' ,'foox' , 'foox&bar=456' , 'bar=456&foox' , 'abc=789&foox&bar=456' ,'foox=' , 'foox=&bar=456' , 'bar=456&foox=' , 'abc=789&foox=&bar=456' ,'foox=123', 'foox=123&bar=456', 'bar=456&foox=123', 'abc=789&foox=123&bar=456' ); @exp = ('' , 'bar=456' , 'bar=456' , 'abc=789&bar=456' ,'' , 'bar=456' , 'bar=456' , 'abc=789&bar=456' ,'' , 'bar=456' , 'bar=456' , 'abc=789&bar=456' ,'xfoo' , 'xfoo&bar=456' , 'bar=456&xfoo' , 'abc=789&xfoo&bar=456' ,'xfoo=' , 'xfoo=&bar=456' , 'bar=456&xfoo=' , 'abc=789&xfoo=&bar=456' ,'xfoo=123', 'xfoo=123&bar=456', 'bar=456&xfoo=123', 'abc=789&xfoo=123&bar=456' ,'foox' , 'foox&bar=456' , 'bar=456&foox' , 'abc=789&foox&bar=456' ,'foox=' , 'foox=&bar=456' , 'bar=456&foox=' , 'abc=789&foox=&bar=456' ,'foox=123', 'foox=123&bar=456', 'bar=456&foox=123', 'abc=789&foox=123&bar=456' ); print "Succ | Input | Output | Expected \n"; print "-----+--------------------------+--------------------------+-------------------------\n"; for($i=0; $i <= $#in; $i++) { $out = $in[$i]; $out =~ s/_PUT_REGEX_HERE_//; $succ = ($out eq $exp[$i] ? 'PASS' : 'FAIL'); #if($succ eq 'FAIL') #{ printf("%s | %- 24s | %- 24s | %- 24s\n", $succ, $in[$i], $out, $exp[$i]); #} }

    Read the article

  • Connection Reset on MySQL query

    - by sunwukung
    OK, I'm flummoxed. I'm trying to execute a query on a database (locally) and I keep getting a connection reset error. I've been using the method below in a generic DAO class to build a query string and pass to Zend_Db API. public function insert($params) { $loop = false; $keys = $values = ''; foreach($params as $k => $v){ if($loop == true){ $keys .= ','; $values .= ','; } $keys .= $this->db->quoteIdentifier($k); $values .= $this->db->quote($v); $loop = true; } $sql = "INSERT INTO " . $this->table_name . " ($keys) VALUES ($values)"; //formatResult returns an array of info regarding the status and any result sets of the query //I've commented that method call out anyway, so I don't think it's that try { $this->db->query($sql); return $this->formatResult(array( true, 'New record inserted into: '.$this->table_name )); }catch(PDOException $e) { return $this->formatResult($e); } } So far, this has worked fine - the errors have been occurring since we generated new tables to record user input. The insert string looks like this: INSERT INTO tablename(`id`,`title`,`summary`,`description`,`keywords`,`type_id`,`categories`) VALUES ('5539','Sample Title','Sample content',' \'Lorem ipsum dolor sit amet, consectetur adipiscing elit. In et pellentesque mauris. Curabitur hendrerit, leo id ultrices pellentesque, est purus mattis ligula, vitae imperdiet neque ligula bibendum sapien. Curabitur aliquet nisi et odio pharetra tincidunt. Phasellus sed iaculis nisl. Fusce commodo mauris et purus vehicula dictum. Nulla feugiat molestie accumsan. Donec fermentum libero in risus tempus elementum aliquam et magna. Fusce vitae sem metus. Aenean commodo pharetra risus, nec pellentesque augue ullamcorper nec. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Nullam vel elit libero. Vestibulum in turpis nunc.\'','this,is,a,sample,array',1,'category title') You'll probably notice the big chunk of whitespace before the Lorem Ipsum string. The description field is being populated from a TinyMCE textarea - I'm guessing it's chucking in some line returns, so I've tried stripping those out. However, even if I disable the TinyMCE field, the reset error still occurs. The next port of call was checking the limits on the table, since it seems to insert if the length of "description" is around the 300 mark (it varies between 310 - 330). The field limit is set to VARCHAR(1500) and the validation on this field won't allow anything past bigger than 1200 with HTML, 800 without. The real kicker is that if I take this sql string and execute it via the command line, it works fine - so I can't for the life of me figure out what's wrong. So, in a nutshell, I'm stumped. Any ideas?

    Read the article

  • MySQL Query GROUP_CONCAT Over Multiple Rows

    - by PeteGO
    I'm getting name and address data out of generic question / answer data to create some kind of normalised reporting database. The query I've got uses group_concat and works for individual sets of questions but not for multiple sets. I've tried to simplify what I'm doing by using just forename and surname and just 3 records, 2 for 1 person and 1 for another. In reality though there are more than 300,000 records. Example of results with qs.Id = 1. QuestionSetId Forename Surname ------------------------------------------------------- 1 Bob Jones Example of results with qs.Id IN (1, 2, 3). QuestionSetId Forename Surname ------------------------------------------------------- 3 Bob,Bob,Frank Jones,Jones,Smith What I would like to see for qs.Id IN (1, 2, 3). QuestionSetId Forename Surname ------------------------------------------------------- 1 Bob Jones 2 Bob Jones 3 Frank Smith So how can I make the 2nd example return a separate row for each set of name and address information? I realise the current way the data is stored is "questionable" but I cannot change the way the data is stored. I can get sets of individual answers but not sure how to combine the others. My simplified Schema that I cannot change: CREATE TABLE StaticQuestion ( Id INT NOT NULL, StaticText VARCHAR(500) NOT NULL); CREATE TABLE Question ( Id INT NOT NULL, Text VARCHAR(500) NOT NULL); CREATE TABLE StaticQuestionQuestionLink ( Id INT NOT NULL, StaticQuestionId INT NOT NULL, QuestionId INT NOT NULL, DateEffective DATETIME NOT NULL); CREATE TABLE Answer ( Id INT NOT NULL, Text VARCHAR(500) NOT NULL); CREATE TABLE QuestionSet ( Id INT NOT NULL, DateEffective DATETIME NOT NULL); CREATE TABLE QuestionAnswerLink ( Id INT NOT NULL, QuestionSetId INT NOT NULL, QuestionId INT NOT NULL, AnswerId INT NOT NULL, StaticQuestionId INT NOT NULL); Some example data for only forename and surname. INSERT INTO StaticQuestion (Id, StaticText) VALUES (1, 'FirstName'), (2, 'LastName'); INSERT INTO Question (Id, Text) VALUES (1, 'What is your first name?'), (2, 'What is your forename?'), (3, 'What is your Surname?'); INSERT INTO StaticQuestionQuestionLink (Id, StaticQuestionId, QuestionId, DateEffective) VALUES (1, 1, 1, '2001-01-01'), (2, 1, 2, '2008-08-08'), (3, 2, 3, '2001-01-01'); INSERT INTO Answer (Id, Text) VALUES (1, 'Bob'), (2, 'Jones'), (3, 'Bob'), (4, 'Jones'), (5, 'Frank'), (6, 'Smith'); INSERT INTO QuestionSet (Id, DateEffective) VALUES (1, '2002-03-25'), (2, '2009-05-05'), (3, '2009-08-06'); INSERT INTO QuestionAnswerLink (Id, QuestionSetId, QuestionId, AnswerId, StaticQuestionId) VALUES (1, 1, 1, 1, 1), (2, 1, 3, 2, 2), (3, 2, 2, 3, 1), (4, 2, 3, 4, 2), (5, 3, 2, 5, 1), (6, 3, 3, 6, 2); Just in case SQLFiddle is down here are the 3 queries from the examples I've linked to: 1: - working query but only on 1 set of data. SELECT MAX(QuestionSetId) AS QuestionSetId, GROUP_CONCAT(Forename) AS Forename, GROUP_CONCAT(Surname) AS Surname FROM (SELECT x.QuestionSetId, CASE x.StaticQuestionId WHEN 1 THEN Text END AS Forename, CASE x.StaticQuestionId WHEN 2 THEN Text END AS Surname FROM (SELECT (SELECT link.StaticQuestionId FROM StaticQuestionQuestionLink link WHERE link.Id = qa.QuestionId AND link.DateEffective <= qs.DateEffective AND link.StaticQuestionId IN (1, 2) ORDER BY link.DateEffective DESC LIMIT 1) AS StaticQuestionId, a.Text, qa.QuestionSetId FROM QuestionSet qs INNER JOIN QuestionAnswerLink qa ON qs.Id = qa.QuestionSetId INNER JOIN Answer a ON qa.AnswerId = a.Id WHERE qs.Id IN (1)) x) y 2: - working query but undesired results on multiple sets of data. SELECT MAX(QuestionSetId) AS QuestionSetId, GROUP_CONCAT(Forename) AS Forename, GROUP_CONCAT(Surname) AS Surname FROM (SELECT x.QuestionSetId, CASE x.StaticQuestionId WHEN 1 THEN Text END AS Forename, CASE x.StaticQuestionId WHEN 2 THEN Text END AS Surname FROM (SELECT (SELECT link.StaticQuestionId FROM StaticQuestionQuestionLink link WHERE link.Id = qa.QuestionId AND link.DateEffective <= qs.DateEffective AND link.StaticQuestionId IN (1, 2) ORDER BY link.DateEffective DESC LIMIT 1) AS StaticQuestionId, a.Text, qa.QuestionSetId FROM QuestionSet qs INNER JOIN QuestionAnswerLink qa ON qs.Id = qa.QuestionSetId INNER JOIN Answer a ON qa.AnswerId = a.Id WHERE qs.Id IN (1, 2, 3)) x) y 3: - working query on multiple sets of data only on 1 field (answer) though. SELECT qs.Id AS QuestionSet, a.Text AS Answer FROM QuestionSet qs INNER JOIN QuestionAnswerLink qalink ON qs.Id = qalink.QuestionSetId INNER JOIN StaticQuestionQuestionLink sqqlink ON qalink.QuestionId = sqqlink.QuestionId INNER JOIN Answer a ON qalink.AnswerId = a.Id WHERE sqqlink.StaticQuestionId = 1 /* FirstName */ AND sqqlink.DateEffective = (SELECT DateEffective FROM StaticQuestionQuestionLink WHERE StaticQuestionId = 1 AND DateEffective <= qs.DateEffective ORDER BY DateEffective DESC LIMIT 1)

    Read the article

  • Improving the performance of XSL

    - by Rachel
    In the below XSL for the variable "insert-data", I have an input param with the structure, <insert-data> <data compareIndex="4" nodeName="d1e1"> <a/> </data> <data compareIndex="5" nodeName="d1e1"> <b/> </data> <data compareIndex="7" nodeName="d1e2"> <a/> </data> <data compareIndex="9" nodeName="d1e2"> <b/> </data> </insert-data> where "nodeName" is the id of a node and "compareIndex" is the position of the text content relative to the node having id "$nodeName". I am using the below XSL to select all the text nodes(generate-id) that satisfy the above condition and construct a data xml. The below implementation works perfectly but the time taken for the execution is in min. Is there a better way of implementing or is there any in-efficient operation being used. From my observation the code where the preceding text length is calculated consumes the major time. Please share your thoughts to improve the performance of the XSL. I am using Java SAXON XSL transformer. <xsl:variable name="insert-data" as="element()*"> <xsl:for-each select="$insert-file/insert-data/data"> <xsl:sort select="xsd:integer(@index)"/> <xsl:variable name="compareIndex" select="xsd:integer(@compareIndex)" /> <xsl:variable name="nodeName" select="@nodeName" /> <xsl:variable name="nodeContent" as="node()"> <xsl:copy-of select="node()"/> </xsl:variable> <xsl:for-each select="$main-root/*//text()[ancestor::*[@id = $nodeName]]"> <xsl:variable name="preTextLength" as="xsd:integer" select="sum((preceding::text())[. ancestor::*[@id = $nodeName]]/string-length(.))" /> <xsl:variable name="currentTextLength" as="xsd:integer" select="string-length(.)" /> <xsl:variable name="sum" select="$preTextLength + $currentTextLength" as="xsd:integer"></xsl:variable> <xsl:variable name="split-index" select="$compareIndex - $preTextLength" as="xsd:integer"></xsl:variable> <xsl:if test="($sum ge $compareIndex) and ($compareIndex gt $preTextLength)"> <data split-index="{$split-index}" text-id="{generate-id(.)}"> <xsl:copy-of select="$nodeContent"/> </data> </xsl:if> </xsl:for-each> </xsl:for-each> </xsl:variable>

    Read the article

  • LINQ Generic Query with inherited base class?

    - by sah302
    I am trying to write some generic LINQ queries for my entities, but am having issue doing the more complex things. Right now I am using an EntityDao class that has all my generics and each of my object class Daos (such as Accomplishments Dao) inherit it, am example: using LCFVB.ObjectsNS; using LCFVB.EntityNS; namespace AccomplishmentNS { public class AccomplishmentDao : EntityDao<Accomplishment>{} } Now my entityDao has the following code: using LCFVB.ObjectsNS; using LCFVB.LinqDataContextNS; namespace EntityNS { public abstract class EntityDao<ImplementationType> where ImplementationType : Entity { public ImplementationType getOneByValueOfProperty(string getProperty, object getValue) { ImplementationType entity = null; if (getProperty != null && getValue != null) { //Nhibernate Example: //ImplementationType entity = default(ImplementationType); //entity = Me.session.CreateCriteria(Of ImplementationType)().Add(Expression.Eq(getProperty, getValue)).UniqueResult(Of InterfaceType)() LCFDataContext lcfdatacontext = new LCFDataContext(); //Generic LINQ Query Here lcfdatacontext.GetTable<ImplementationType>(); lcfdatacontext.SubmitChanges(); lcfdatacontext.Dispose(); } return entity; } public bool insertRow(ImplementationType entity) { if (entity != null) { //Nhibernate Example: //Me.session.Save(entity, entity.Id) //Me.session.Flush() LCFDataContext lcfdatacontext = new LCFDataContext(); //Generic LINQ Query Here lcfdatacontext.GetTable<ImplementationType>().InsertOnSubmit(entity); lcfdatacontext.SubmitChanges(); lcfdatacontext.Dispose(); return true; } else { return false; } } } }             I have gotten the insertRow function working, however I am not even sure how to go about doing getOnebyValueOfProperty, the closest thing I could find on this site was: http://stackoverflow.com/questions/2157560/generic-linq-to-sql-query How can I pass in the column name and the value I am checking against generically using my current set-up? It seems like from that link it's impossible since using a where predicate because entity class doesn't know what any of the properties are until I pass them in. Lastly, I need some way of setting a new object as the return type set to the implementation type, in nhibernate (what I am trying to convert from) it was simply this line that did it: ImplentationType entity = default(ImplentationType); However default is an nhibernate command, how would I do this for LINQ? EDIT: getOne doesn't seem to work even when just going off the base class (this is a partial class of the auto generated LINQ classes). I even removed the generics. I tried: namespace ObjectsNS { public partial class Accomplishment { public Accomplishment getOneByWhereClause(Expression<Action<Accomplishment, bool>> singleOrDefaultClause) { Accomplishment entity = new Accomplishment(); if (singleOrDefaultClause != null) { LCFDataContext lcfdatacontext = new LCFDataContext(); //Generic LINQ Query Here entity = lcfdatacontext.Accomplishments.SingleOrDefault(singleOrDefaultClause); lcfdatacontext.Dispose(); } return entity; } } } Get the following error: Error 1 Overload resolution failed because no accessible 'SingleOrDefault' can be called with these arguments: Extension method 'Public Function SingleOrDefault(predicate As System.Linq.Expressions.Expression(Of System.Func(Of Accomplishment, Boolean))) As Accomplishment' defined in 'System.Linq.Queryable': Value of type 'System.Action(Of System.Func(Of LCFVB.ObjectsNS.Accomplishment, Boolean))' cannot be converted to 'System.Linq.Expressions.Expression(Of System.Func(Of LCFVB.ObjectsNS.Accomplishment, Boolean))'. Extension method 'Public Function SingleOrDefault(predicate As System.Func(Of Accomplishment, Boolean)) As Accomplishment' defined in 'System.Linq.Enumerable': Value of type 'System.Action(Of System.Func(Of LCFVB.ObjectsNS.Accomplishment, Boolean))' cannot be converted to 'System.Func(Of LCFVB.ObjectsNS.Accomplishment, Boolean)'. 14 LCF Okay no problem I changed: public Accomplishment getOneByWhereClause(Expression<Action<Accomplishment, bool>> singleOrDefaultClause) to: public Accomplishment getOneByWhereClause(Expression<Func<Accomplishment, bool>> singleOrDefaultClause) Error goes away. Alright, but now when I try to call the method via: Accomplishment accomplishment = new Accomplishment(); var result = accomplishment.getOneByWhereClause(x=>x.Id = 4) It doesn't work it says x is not declared. I also tried getOne, and various other Expression =(

    Read the article

  • HttpClient multithread performance

    - by pepper
    I have an application which downloads more than 4500 html pages from 62 target hosts using HttpClient (4.1.3 or 4.2-beta). It runs on Windows 7 64-bit. Processor - Core i7 2600K. Network bandwidth - 54 Mb/s. At this moment it uses such parameters: DefaultHttpClient and PoolingClientConnectionManager; Also it hasIdleConnectionMonitorThread from http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html; Maximum total connections = 80; Default maximum connections per route = 5; For thread management it uses ForkJoinPool with the parallelism level = 5 (Do I understand correctly that it is a number of working threads?) In this case my network usage (in Windows task manager) does not rise above 2.5%. To download 4500 pages it takes 70 minutes. And in HttpClient logs I have such things: DEBUG ForkJoinPool-2-worker-1 [org.apache.http.impl.conn.PoolingClientConnectionManager]: Connection released: [id: 209][route: {}-http://stackoverflow.com][total kept alive: 6; route allocated: 1 of 5; total allocated: 10 of 80] Total allocated connections do not raise above 10-12, in spite of that I've set it up to 80 connections. If I'll try to rise parallelism level to 20 or 80, network usage remains the same but a lot connection time-outs will be generated. I've read tutorials on hc.apache.org (HttpClient Performance Optimization Guide and HttpClient Threading Guide) but they does not help. Task's code looks like this: public class ContentDownloader extends RecursiveAction { private final HttpClient httpClient; private final HttpContext context; private List<Entry> entries; public ContentDownloader(HttpClient httpClient, List<Entry> entries){ this.httpClient = httpClient; context = new BasicHttpContext(); this.entries = entries; } private void computeDirectly(Entry entry){ final HttpGet get = new HttpGet(entry.getLink()); try { HttpResponse response = httpClient.execute(get, context); int statusCode = response.getStatusLine().getStatusCode(); if ( (statusCode >= 400) && (statusCode <= 600) ) { logger.error("Couldn't get content from " + get.getURI().toString() + "\n" + response.toString()); } else { HttpEntity entity = response.getEntity(); if (entity != null) { String htmlContent = EntityUtils.toString(entity).trim(); entry.setHtml(htmlContent); EntityUtils.consumeQuietly(entity); } } } catch (Exception e) { } finally { get.releaseConnection(); } } @Override protected void compute() { if (entries.size() <= 1){ computeDirectly(entries.get(0)); return; } int split = entries.size() / 2; invokeAll(new ContentDownloader(httpClient, entries.subList(0, split)), new ContentDownloader(httpClient, entries.subList(split, entries.size()))); } } And the question is - what is the best practice to use multi threaded HttpClient, may be there is a some rules for setting up ConnectionManager and HttpClient? How can I use all of 80 connections and raise network usage? If necessary, I will provide more code.

    Read the article

  • Rails ActiveRecord friendly code from a Complex Join, Sum, and Group query

    - by Chad M
    PROBLEM Hello, I am having no luck trying to break down this SQL statement into ActiveRecord/Rails friendly code and I'd like to learn how I can avoid a find_by_sql statement in this situation. Scenario I have users that create audits when they perform an action. Each audit is of a specific audit_activity. Each audit_activity is worth a certain number of points, based on score_weight. I need to find the total scores of each user, based on their total accumulated audit_activity score_weights. Eventually I'll need to rank them which means adding a sort to this as well. My Code Here is my sql and simplified versions of the tables in question. Any thoughts? SQL with full column names (for clarity) SELECT users.id, u.email, SUM(audit_activity.score_weight) FROM users JOIN audits ON users.id = audits.user_id JOIN audit_activities ON audit_activities.id = audits.audit_activity_id GROUP BY users.id; Models: User, Audit, AuditActivity User fields: id, email class User < ActiveRecord::Base include Clearance::User has_many :audits end Audit fields: id, user_id, audit_activity_id class Audit < ActiveRecord::Base belongs_to :user belongs_to :audit_activity end AuditActivity fields: id, score_weight class AuditActivity < ActiveRecord::Base has_many :audits end Example Data Here is a set of SQL statements so you can play with similar data I'm working with and see what comes up when the concerned query is run. You should just be able to copy/paste the whole thing into a database query browser. CREATE TABLE users( id INTEGER NOT NULL, email TEXT (25), PRIMARY KEY (id) ); CREATE TABLE audits( id INTEGER NOT NULL, user_id INTEGER, audit_activity_id INTEGER, PRIMARY KEY (id) ); CREATE TABLE audit_activities( id INTEGER NOT NULL, score_weight INTEGER, PRIMARY KEY (id) ); INSERT INTO users(id, email) VALUES(1, "[email protected]"); INSERT INTO users(id, email) VALUES(2, "[email protected]"); INSERT INTO users(id, email) VALUES(3, "[email protected]"); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(1, 1, 1); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(2, 1, 2); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(3, 1, 1); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(4, 1, 3); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(5, 1, 1); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(6, 1, 4); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(7, 2, 4); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(8, 2, 4); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(9, 2, 4); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(10, 3, 3); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(11, 3, 2); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(12, 3, 2); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(13, 3, 2); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(14, 3, 3); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(15, 3, 1); INSERT INTO audits(id, user_id, audit_activity_id) VALUES(16, 3, 1); INSERT INTO audit_activities(id, score_weight) VALUES(1, 1); INSERT INTO audit_activities(id, score_weight) VALUES(2, 2); INSERT INTO audit_activities(id, score_weight) VALUES(3, 7); INSERT INTO audit_activities(id, score_weight) VALUES(4, 11); The Query Again, here is the query. SELECT u.id, u.email, SUM(aa.score_weight) FROM users u JOIN audits a ON u.id = a.user_id JOIN audit_activities aa ON aa.id = a.audit_activity_id GROUP BY u.id; Many Thanks, Chad

    Read the article

  • Optimizing python code performance when importing zipped csv to a mongo collection

    - by mark
    I need to import a zipped csv into a mongo collection, but there is a catch - every record contains a timestamp in Pacific Time, which must be converted to the local time corresponding to the (longitude,latitude) pair found in the same record. The code looks like so: def read_csv_zip(path, timezones): with ZipFile(path) as z, z.open(z.namelist()[0]) as input: csv_rows = csv.reader(input) header = csv_rows.next() check,converters = get_aux_stuff(header) for csv_row in csv_rows: if check(csv_row): row = { converter[0]:converter[1](value) for converter, value in zip(converters, csv_row) if allow_field(converter) } ts = row['ts'] lng, lat = row['loc'] found_tz_entry = timezones.find_one(SON({'loc': {'$within': {'$box': [[lng-tz_lookup_radius, lat-tz_lookup_radius],[lng+tz_lookup_radius, lat+tz_lookup_radius]]}}})) if found_tz_entry: tz_name = found_tz_entry['tz'] local_ts = ts.astimezone(timezone(tz_name)).replace(tzinfo=None) row['tz'] = tz_name else: local_ts = (ts.astimezone(utc) + timedelta(hours = int(lng/15))).replace(tzinfo = None) row['local_ts'] = local_ts yield row def insert_documents(collection, source, batch_size): while True: items = list(itertools.islice(source, batch_size)) if len(items) == 0: break; try: collection.insert(items) except: for item in items: try: collection.insert(item) except Exception as exc: print("Failed to insert record {0} - {1}".format(item['_id'], exc)) def main(zip_path): with Connection() as connection: data = connection.mydb.data timezones = connection.timezones.data insert_documents(data, read_csv_zip(zip_path, timezones), 1000) The code proceeds as follows: Every record read from the csv is checked and converted to a dictionary, where some fields may be skipped, some titles be renamed (from those appearing in the csv header), some values may be converted (to datetime, to integers, to floats. etc ...) For each record read from the csv, a lookup is made into the timezones collection to map the record location to the respective time zone. If the mapping is successful - that timezone is used to convert the record timestamp (pacific time) to the respective local timestamp. If no mapping is found - a rough approximation is calculated. The timezones collection is appropriately indexed, of course - calling explain() confirms it. The process is slow. Naturally, having to query the timezones collection for every record kills the performance. I am looking for advises on how to improve it. Thanks. EDIT The timezones collection contains 8176040 records, each containing four values: > db.data.findOne() { "_id" : 3038814, "loc" : [ 1.48333, 42.5 ], "tz" : "Europe/Andorra" } EDIT2 OK, I have compiled a release build of http://toblerity.github.com/rtree/ and configured the rtree package. Then I have created an rtree dat/idx pair of files corresponding to my timezones collection. So, instead of calling collection.find_one I call index.intersection. Surprisingly, not only there is no improvement, but it works even more slowly now! May be rtree could be fine tuned to load the entire dat/idx pair into RAM (704M), but I do not know how to do it. Until then, it is not an alternative. In general, I think the solution should involve parallelization of the task. EDIT3 Profile output when using collection.find_one: >>> p.sort_stats('cumulative').print_stats(10) Tue Apr 10 14:28:39 2012 ImportDataIntoMongo.profile 64549590 function calls (64549180 primitive calls) in 1231.257 seconds Ordered by: cumulative time List reduced from 730 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.012 0.012 1231.257 1231.257 ImportDataIntoMongo.py:1(<module>) 1 0.001 0.001 1230.959 1230.959 ImportDataIntoMongo.py:187(main) 1 853.558 853.558 853.558 853.558 {raw_input} 1 0.598 0.598 370.510 370.510 ImportDataIntoMongo.py:165(insert_documents) 343407 9.965 0.000 359.034 0.001 ImportDataIntoMongo.py:137(read_csv_zip) 343408 2.927 0.000 287.035 0.001 c:\python27\lib\site-packages\pymongo\collection.py:489(find_one) 343408 1.842 0.000 274.803 0.001 c:\python27\lib\site-packages\pymongo\cursor.py:699(next) 343408 2.542 0.000 271.212 0.001 c:\python27\lib\site-packages\pymongo\cursor.py:644(_refresh) 343408 4.512 0.000 253.673 0.001 c:\python27\lib\site-packages\pymongo\cursor.py:605(__send_message) 343408 0.971 0.000 242.078 0.001 c:\python27\lib\site-packages\pymongo\connection.py:871(_send_message_with_response) Profile output when using index.intersection: >>> p.sort_stats('cumulative').print_stats(10) Wed Apr 11 16:21:31 2012 ImportDataIntoMongo.profile 41542960 function calls (41542536 primitive calls) in 2889.164 seconds Ordered by: cumulative time List reduced from 778 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.028 0.028 2889.164 2889.164 ImportDataIntoMongo.py:1(<module>) 1 0.017 0.017 2888.679 2888.679 ImportDataIntoMongo.py:202(main) 1 2365.526 2365.526 2365.526 2365.526 {raw_input} 1 0.766 0.766 502.817 502.817 ImportDataIntoMongo.py:180(insert_documents) 343407 9.147 0.000 491.433 0.001 ImportDataIntoMongo.py:152(read_csv_zip) 343406 0.571 0.000 391.394 0.001 c:\python27\lib\site-packages\rtree-0.7.0-py2.7.egg\rtree\index.py:384(intersection) 343406 379.957 0.001 390.824 0.001 c:\python27\lib\site-packages\rtree-0.7.0-py2.7.egg\rtree\index.py:435(_intersection_obj) 686513 22.616 0.000 38.705 0.000 c:\python27\lib\site-packages\rtree-0.7.0-py2.7.egg\rtree\index.py:451(_get_objects) 343406 6.134 0.000 33.326 0.000 ImportDataIntoMongo.py:162(<dictcomp>) 346 0.396 0.001 30.665 0.089 c:\python27\lib\site-packages\pymongo\collection.py:240(insert) EDIT4 I have parallelized the code, but the results are still not very encouraging. I am convinced it could be done better. See my own answer to this question for details.

    Read the article

  • ListView slow performance

    - by Mohamed Hemdan
    I've created a list of recipes using Listview/customcursoradapter. A custom layout includes a photo for the recipe , Now I've some problems with the performance of viewing and scrolling the Listview although it has only 10 records (Target is 150). sometimes i get this error java.lang.OutOfMemoryError: bitmap size exceeds VM budget , I've tried to implement the Async task but i failed to do it. Is there any way i can overcome this problem? Your help is highly appreciated !! Here is my GetView method public View getView(int position, View convertView, ViewGroup parent) { View row = super.getView(position, convertView, parent); Cursor cursbbn = getCursor(); if (row == null) { LayoutInflater inflater = (LayoutInflater) localContext.getSystemService(Context.LAYOUT_INFLATER_SERVICE); row = inflater.inflate(R.layout.listtype, null); } String Title = cursbbn.getString(2); String SandID=cursbbn.getString(1); String Readyin = cursbbn.getString(4); String Faovoites=cursbbn.getString(8); TextView titler=(TextView)row.findViewById(R.id.listmaintitle); TextView readyinr=(TextView)row.findViewById(R.id.listreadyin); int colorPos = position % colors.length; row.setBackgroundColor(colors[colorPos]); titler.setText(Title); readyinr.setText(Readyin); ImageView picture = (ImageView) row.findViewById(R.id.imageView1); Bitmap bitImg1 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0001); Bitmap bitImg2 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0002); Bitmap bitImg3 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0003); Bitmap bitImg4 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0004); Bitmap bitImg5 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0005); Bitmap bitImg6 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0006); Bitmap bitImg7 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0007); Bitmap bitImg8 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0008); Bitmap bitImg9 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0009); Bitmap bitImg10 = BitmapFactory.decodeResource(localContext.getResources(), R.drawable.rec0010); if(SandID.contentEquals("0001")) picture.setImageBitmap(getRoundedCornerImage(bitImg1)); if(SandID.contentEquals("0002")) picture.setImageBitmap(getRoundedCornerImage(bitImg2)); if(SandID.contentEquals("0003")) picture.setImageBitmap(getRoundedCornerImage(bitImg3)); if(SandID.contentEquals("0004")) picture.setImageBitmap(getRoundedCornerImage(bitImg4)); if(SandID.contentEquals("0005")) picture.setImageBitmap(getRoundedCornerImage(bitImg5)); if(SandID.contentEquals("0006")) picture.setImageBitmap(getRoundedCornerImage(bitImg6)); if(SandID.contentEquals("0007")) picture.setImageBitmap(getRoundedCornerImage(bitImg7)); if(SandID.contentEquals("0008")) picture.setImageBitmap(getRoundedCornerImage(bitImg8)); if(SandID.contentEquals("0009")) picture.setImageBitmap(getRoundedCornerImage(bitImg9)); if(SandID.contentEquals("0010")) picture.setImageBitmap(getRoundedCornerImage(bitImg10)); return row; } And This is the error : 05-02 03:11:55.898: E/AndroidRuntime(376): FATAL EXCEPTION: main 05-02 03:11:55.898: E/AndroidRuntime(376): java.lang.OutOfMemoryError: bitmap size exceeds VM budget 05-02 03:11:55.898: E/AndroidRuntime(376): at android.graphics.BitmapFactory.nativeDecodeAsset(Native Method) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.graphics.BitmapFactory.decodeStream(BitmapFactory.java:460) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.graphics.BitmapFactory.decodeResourceStream(BitmapFactory.java:336) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.graphics.BitmapFactory.decodeResource(BitmapFactory.java:359) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.graphics.BitmapFactory.decodeResource(BitmapFactory.java:385) 05-02 03:11:55.898: E/AndroidRuntime(376): at master.chef.mediamaster.AlternateRowCursorAdapter.getView(AlternateRowCursorAdapter.java:83) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.AbsListView.obtainView(AbsListView.java:1409) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.ListView.makeAndAddView(ListView.java:1745) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.ListView.fillUp(ListView.java:700) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.ListView.fillGap(ListView.java:646) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.AbsListView.trackMotionScroll(AbsListView.java:3399) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.AbsListView.onTouchEvent(AbsListView.java:2233) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.widget.ListView.onTouchEvent(ListView.java:3446) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.View.dispatchTouchEvent(View.java:3885) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:903) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:942) 05-02 03:11:55.898: E/AndroidRuntime(376): at com.android.internal.policy.impl.PhoneWindow$DecorView.superDispatchTouchEvent(PhoneWindow.java:1691) 05-02 03:11:55.898: E/AndroidRuntime(376): at com.android.internal.policy.impl.PhoneWindow.superDispatchTouchEvent(PhoneWindow.java:1125) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.app.Activity.dispatchTouchEvent(Activity.java:2096) 05-02 03:11:55.898: E/AndroidRuntime(376): at com.android.internal.policy.impl.PhoneWindow$DecorView.dispatchTouchEvent(PhoneWindow.java:1675) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewRoot.deliverPointerEvent(ViewRoot.java:2194) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.view.ViewRoot.handleMessage(ViewRoot.java:1878) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.os.Handler.dispatchMessage(Handler.java:99) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.os.Looper.loop(Looper.java:123) 05-02 03:11:55.898: E/AndroidRuntime(376): at android.app.ActivityThread.main(ActivityThread.java:3683) 05-02 03:11:55.898: E/AndroidRuntime(376): at java.lang.reflect.Method.invokeNative(Native Method) 05-02 03:11:55.898: E/AndroidRuntime(376): at java.lang.reflect.Method.invoke(Method.java:507) 05-02 03:11:55.898: E/AndroidRuntime(376): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:839) 05-02 03:11:55.898: E/AndroidRuntime(376): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:597) 05-02 03:11:55.898: E/AndroidRuntime(376): at dalvik.system.NativeStart.main(Native Method)

    Read the article

  • Oracle performance problem

    - by jreid42
    We are using an Oracle 11G machine that is very powerful; has redundant storage etc. It's a beast from what I have been told. We just got this DB for a tool that when I first came on as a coop had like 20 people using, now its upwards of 150 people. I am the only one working on it :( We currently have a system in place that distributes PERL scripts across our entire data center essentially giving us a sort of "grid" computing power. The Perl scripts run a sort of simulation and report back the results to the database. They do selects / inserts. The load is not very high for each script but it could be happening across 20-50 systems at the same time. We then have multiple data centers and users all hitting the same database with this same approach. Our main problem with this is that our database is getting overloaded with connections and having to drop some. We sometimes have upwards of 500 connections. These are old perl scripts and they do not handle this well. Essentially they fail and the results are lost. I would rather avoid having to rewrite a lot of these as they are poorly written, and are a headache to even look at. The database itself is not overloaded, just the connection overhead is too high. We open a connection, make a quick query and then drop the connection. Very short connections but many of them. The database team has basically said we need to lower the number of connections or they are going to ignore us. Because this is distributed across our farm we cant implement persistent connections. I do this with our webserver; but its on a fixed system. The other ones are perl scripts that get opened and closed by the distribution tool and thus arent always running. What would be my best approach to resolving this issue? The scripts themselves can wait for a connection to be open. They do not need to act immediately. Some sort of queing system? I've been suggested to set up a few instances of a tool called "SQL Relay". Maybe one in each data center. How reliable is this tool? How good is this approach? Would it work for what we need? We could have one for each data center and relay requests through it to our main database, keeping a pipeline of open persistent connections? Does this make sense? Is there any other suggestions you can make? Any ideas? Any help would be greatly appreciated. Sadly I am just a coop student working for a very big company and somehow all of this has landed all on my shoulders (there is literally nobody to ask for help; its a hardware company, everybody is hardware engineers, and the database team is useless and in India) and I am quite lost as what the best approach would be? I am extremely overworked and this problem is interfering with on going progress and basically needs to be resolved as quickly as possible; preferably without rewriting the whole system, purchasing hardware (not gonna happen), or shooting myself in the foot. HELP LOL!

    Read the article

  • How optimize code with introspection + heavy alloc on iPhone

    - by mamcx
    I have a problem. I try to display a UITable that could have 2000-20000 records (typicall numbers.) I have a SQLite database similar to the Apple contacts application. I do all the tricks I know to get a smoth scroll, but I have a problem. I load the data in 50 recods blocks. Then, when the user scroll, request next 50 until finish the list. However, load that 50 records cause a notable "pause" in loading and scrolling. Everything else works fine. I cache the data, have opaque cells, draw it by code, etc... I swap the code loading the same data in dicts and have a performance boost but wonder if I could keep my object oriented aproach and improve the actual code. This is the code I think have the performance problem: -(NSArray *) loadAndFill: (NSString *)sql theClass: (Class)cls { [self openDb]; NSMutableArray *list = [NSMutableArray array]; NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; DbObject *ds; Class myClass = NSClassFromString([DbObject getTableName:cls]); FMResultSet *rs = [self load:sql]; while ([rs next]) { ds = [[myClass alloc] init]; NSDictionary *props = [ds properties]; NSString *fieldType = nil; id fieldValue; for (NSString *fieldName in [props allKeys]) { fieldType = [props objectForKey: fieldName]; fieldValue = [self ValueForField:rs Name:fieldName Type:fieldType]; [ds setValue:fieldValue forKey:fieldName]; } [list addObject :ds]; [ds release]; } [rs close]; [pool drain]; return list; } And I think the main culprit is: -(id) ValueForField: (FMResultSet *)rs Name:(NSString *)fieldName Type:(NSString *)fieldType { id fieldValue = nil; if ([fieldType isEqualToString:@"i"] || // int [fieldType isEqualToString:@"I"] || // unsigned int [fieldType isEqualToString:@"s"] || // short [fieldType isEqualToString:@"S"] || // unsigned short [fieldType isEqualToString:@"f"] || // float [fieldType isEqualToString:@"d"] ) // double { fieldValue = [NSNumber numberWithInt: [rs longForColumn:fieldName]]; } else if ([fieldType isEqualToString:@"B"]) // bool or _Bool { fieldValue = [NSNumber numberWithBool: [rs boolForColumn:fieldName]]; } else if ([fieldType isEqualToString:@"l"] || // long [fieldType isEqualToString:@"L"] || // usigned long [fieldType isEqualToString:@"q"] || // long long [fieldType isEqualToString:@"Q"] ) // unsigned long long { fieldValue = [NSNumber numberWithLong: [rs longForColumn:fieldName]]; } else if ([fieldType isEqualToString:@"c"] || // char [fieldType isEqualToString:@"C"] ) // unsigned char { fieldValue = [rs stringForColumn:fieldName]; //Is really a boolean? if ([fieldValue isEqualToString:@"0"] || [fieldValue isEqualToString:@"1"]) { fieldValue = [NSNumber numberWithInt: [fieldValue intValue]]; } } else if ([fieldType hasPrefix:@"@"] ) // Object { NSString *className = [fieldType substringWithRange:NSMakeRange(2, [fieldType length]-3)]; if ([className isEqualToString:@"NSString"]) { fieldValue = [rs stringForColumn:fieldName]; } else if ([className isEqualToString:@"NSDate"]) { NSDateFormatter* dateFormatter = [[NSDateFormatter alloc] init]; [dateFormatter setDateFormat:@"yyyy-MM-dd'T'HH:mm:ss"]; NSString *theDate = [rs stringForColumn:fieldName]; if (theDate) { fieldValue = [dateFormatter dateFromString: theDate]; } else { fieldValue = nil; } [dateFormatter release]; } else if ([className isEqualToString:@"NSInteger"]) { fieldValue = [NSNumber numberWithInt: [rs intForColumn :fieldName]]; } else if ([className isEqualToString:@"NSDecimalNumber"]) { fieldValue = [rs stringForColumn :fieldName]; if (fieldValue) { fieldValue = [NSDecimalNumber decimalNumberWithString:[rs stringForColumn :fieldName]]; } } else if ([className isEqualToString:@"NSNumber"]) { fieldValue = [NSNumber numberWithDouble: [rs doubleForColumn:fieldName]]; } else { //Is a relationship one-to-one? if (![fieldType hasPrefix:@"NS"]) { id rel = class_createInstance(NSClassFromString(className), sizeof(unsigned)); Class theClass = [rel class]; if ([rel isKindOfClass:[DbObject class]]) { fieldValue = [rel init]; //Load the record... NSInteger Id = [rs intForColumn:[theClass relationName]]; if (Id>0) { [fieldValue release]; Db *db = [Db currentDb]; fieldValue = [db loadById: theClass theId:Id]; } } } else { NSString *error = [NSString stringWithFormat:@"Err Can't get value for field %@ of type %@", fieldName, fieldType]; NSLog(error); NSException *e = [NSException exceptionWithName:@"DBError" reason:error userInfo:nil]; @throw e; } } } return fieldValue; }

    Read the article

  • Using UUIDs for cheap equals() and hashCode()

    - by Tom McIntyre
    I have an immutable class, TokenList, which consists of a list of Token objects, which are also immutable: @Immutable public final class TokenList { private final List<Token> tokens; public TokenList(List<Token> tokens) { this.tokens = Collections.unmodifiableList(new ArrayList(tokens)); } public List<Token> getTokens() { return tokens; } } I do several operations on these TokenLists that take multiple TokenLists as inputs and return a single TokenList as the output. There can be arbitrarily many TokenLists going in, and each can have arbitrarily many Tokens. These operations are expensive, and there is a good chance that the same operation (ie the same inputs) will be performed multiple times, so I would like to cache the outputs. However, performance is critical, and I am worried about the expense of performing hashCode() and equals() on these objects that may contain arbitrarily many elements (as they are immutable then hashCode could be cached, but equals will still be expensive). This led me to wondering whether I could use a UUID to provide equals() and hashCode() simply and cheaply by making the following updates to TokenList: @Immutable public final class TokenList { private final List<Token> tokens; private final UUID uuid; public TokenList(List<Token> tokens) { this.tokens = Collections.unmodifiableList(new ArrayList(tokens)); this.uuid = UUID.randomUUID(); } public List<Token> getTokens() { return tokens; } public UUID getUuid() { return uuid; } } And something like this to act as a cache key: @Immutable public final class TopicListCacheKey { private final UUID[] uuids; public TopicListCacheKey(TopicList... topicLists) { uuids = new UUID[topicLists.length]; for (int i = 0; i < uuids.length; i++) { uuids[i] = topicLists[i].getUuid(); } } @Override public int hashCode() { return Arrays.hashCode(uuids); } @Override public boolean equals(Object other) { if (other == this) return true; if (other instanceof TopicListCacheKey) return Arrays.equals(uuids, ((TopicListCacheKey) other).uuids); return false; } } I figure that there are 2^128 different UUIDs and I will probably have at most around 1,000,000 TokenList objects active in the application at any time. Given this, and the fact that the UUIDs are used combinatorially in cache keys, it seems that the chances of this producing the wrong result are vanishingly small. Nevertheless, I feel uneasy about going ahead with it as it just feels 'dirty'. Are there any reasons I should not use this system? Will the performance costs of the SecureRandom used by UUID.randomUUID() outweigh the gains (especially since I expect multiple threads to be doing this at the same time)? Are collisions going to be more likely than I think? Basically, is there anything wrong with doing it this way?? Thanks.

    Read the article

  • Make c# matrix code faster

    - by Wam
    Hi all, Working on some matrix code, I'm concerned of performance issues. here's how it works : I've a IMatrix abstract class (with all matrices operations etc), implemented by a ColumnMatrix class. abstract class IMatrix { public int Rows {get;set;} public int Columns {get;set;} public abstract float At(int row, int column); } class ColumnMatrix : IMatrix { private data[]; public override float At(int row, int column) { return data[row + columns * this.Rows]; } } This class is used a lot across my application, but I'm concerned with performance issues. Testing only read for a 2000000x15 matrix against a jagged array of the same size, I get 1359ms for array access agains 9234ms for matrix access : public void TestAccess() { int iterations = 10; int rows = 2000000; int columns = 15; ColumnMatrix matrix = new ColumnMatrix(rows, columns); for (int i = 0; i < rows; i++) for (int j = 0; j < columns; j++) matrix[i, j] = i + j; float[][] equivalentArray = matrix.ToRowsArray(); TimeSpan totalMatrix = new TimeSpan(0); TimeSpan totalArray = new TimeSpan(0); float total = 0f; for (int iteration = 0; iteration < iterations; iteration++) { total = 0f; DateTime start = DateTime.Now; for (int i = 0; i < rows; i++) for (int j = 0; j < columns; j++) total = matrix.At(i, j); totalMatrix += (DateTime.Now - start); total += 1f; //Ensure total is read at least once. total = total > 0 ? 0f : 0f; start = DateTime.Now; for (int i = 0; i < rows; i++) for (int j = 0; j < columns; j++) total = equivalentArray[i][j]; totalArray += (DateTime.Now - start); } if (total < 0f) logger.Info("Nothing here, just make sure we read total at least once."); logger.InfoFormat("Average time for a {0}x{1} access, matrix : {2}ms", rows, columns, totalMatrix.TotalMilliseconds); logger.InfoFormat("Average time for a {0}x{1} access, array : {2}ms", rows, columns, totalArray.TotalMilliseconds); Assert.IsTrue(true); } So my question : how can I make this thing faster ? Is there any way I can make my ColumnMatrix.At faster ? Cheers !

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 234 235 236 237 238 239 240 241 242 243 244 245  | Next Page >