Search Results

Search found 991 results on 40 pages for 'indexed'.

Page 40/40 | < Previous Page | 36 37 38 39 40 

• What statistics can be maintained for a set of numerical data without iterating?

  - by Dan Tao

  Update Just for future reference, I'm going to list all of the statistics that I'm aware of that can be maintained in a rolling collection, recalculated as an O(1) operation on every addition/removal (this is really how I should've worded the question from the beginning): Obvious Count Sum Mean Max* Min* Median** Less Obvious Variance Standard Deviation Skewness Kurtosis Mode*** Weighted Average Weighted Moving Average**** OK, so to put it more accurately: these are not "all" of the statistics I'm aware of. They're just the ones that I can remember off the top of my head right now. *Can be recalculated in O(1) for additions only, or for additions and removals if the collection is sorted (but in this case, insertion is not O(1)). Removals potentially incur an O(n) recalculation for non-sorted collections. **Recalculated in O(1) for a sorted, indexed collection only. ***Requires a fairly complex data structure to recalculate in O(1). ****This can certainly be achieved in O(1) for additions and removals when the weights are assigned in a linearly descending fashion. In other scenarios, I'm not sure. Original Question Say I maintain a collection of numerical data -- let's say, just a bunch of numbers. For this data, there are loads of calculated values that might be of interest; one example would be the sum. To get the sum of all this data, I could... Option 1: Iterate through the collection, adding all the values: double sum = 0.0; for (int i = 0; i < values.Count; i++) sum += values[i]; Option 2: Maintain the sum, eliminating the need to ever iterate over the collection just to find the sum: void Add(double value) { values.Add(value); sum += value; } void Remove(double value) { values.Remove(value); sum -= value; } EDIT: To put this question in more relatable terms, let's compare the two options above to a (sort of) real-world situation: Suppose I start listing numbers out loud and ask you to keep them in your head. I start by saying, "11, 16, 13, 12." If you've just been remembering the numbers themselves and nothing more, and then I say, "What's the sum?", you'd have to think to yourself, "OK, what's 11 + 16 + 13 + 12?" before responding, "52." If, on the other hand, you had been keeping track of the sum yourself while I was listing the numbers (i.e., when I said, "11" you thought "11", when I said "16", you thought, "27," and so on), you could answer "52" right away. Then if I say, "OK, now forget the number 16," if you've been keeping track of the sum inside your head you can simply take 16 away from 52 and know that the new sum is 36, rather than taking 16 off the list and them summing up 11 + 13 + 12. So my question is, what other calculations, other than the obvious ones like sum and average, are like this? SECOND EDIT: As an arbitrary example of a statistic that (I'm almost certain) does require iteration -- and therefore cannot be maintained as simply as a sum or average -- consider if I asked you, "how many numbers in this collection are divisible by the min?" Let's say the numbers are 5, 15, 19, 20, 21, 25, and 30. The min of this set is 5, which divides into 5, 15, 20, 25, and 30 (but not 19 or 21), so the answer is 5. Now if I remove 5 from the collection and ask the same question, the answer is now 2, since only 15 and 30 are divisible by the new min of 15; but, as far as I can tell, you cannot know this without going through the collection again. So I think this gets to the heart of my question: if we can divide kinds of statistics into these categories, those that are maintainable (my own term, maybe there's a more official one somewhere) versus those that require iteration to compute any time a collection is changed, what are all the maintainable ones? What I am asking about is not strictly the same as an online algorithm (though I sincerely thank those of you who introduced me to that concept). An online algorithm can begin its work without having even seen all of the input data; the maintainable statistics I am seeking will certainly have seen all the data, they just don't need to reiterate through it over and over again whenever it changes.

  Read the article

• Need help on Coda slider tabs to move inside an overflow:hidden div

  - by Reden

  I'm not too good at javascript. and hope I can get a bit of help. I'm using Coda Slider 2.0, and have designed it to where the tabs are another slider to the right of the main slider, and each item. Basically like this mootools plugin http://landofcoder.com/demo/mootool/lofslidernews/index2.1.html Problem is the items will not scroll. How do I make the items (or tabs to the right) scroll down as the slider rotates? Otherwise the slider will show the 4th slide but not scroll to the 4th item on the right, but Thanks everyone. Here is the Coda-Slider plugin: // when the DOM is ready... $(document).ready(function () { var $panels = $('#slider .scrollContainer > div'); var $container = $('#slider .scrollContainer'); // if false, we'll float all the panels left and fix the width // of the container var horizontal = true; // float the panels left if we're going horizontal if (horizontal) { $panels.css({ 'float' : 'left', 'position' : 'relative' // IE fix to ensure overflow is hidden }); // calculate a new width for the container (so it holds all panels) $container.css('width', $panels[0].offsetWidth * $panels.length); } // collect the scroll object, at the same time apply the hidden overflow // to remove the default scrollbars that will appear var $scroll = $('#slider .scroll').css('overflow', 'hidden'); // apply our left + right buttons $scroll .before('<img class="scrollButtons left" src="images/scroll_left.png" />') .after('<img class="scrollButtons right" src="images/scroll_right.png" />'); // handle nav selection function selectNav() { $(this) .parents('ul:first') .find('a') .removeClass('selected') .end() .end() .addClass('selected'); } $('#slider .navigation').find('a').click(selectNav); // go find the navigation link that has this target and select the nav function trigger(data) { var el = $('#slider .navigation').find('a[href$="' + data.id + '"]').get(0); selectNav.call(el); } if (window.location.hash) { trigger({ id : window.location.hash.substr(1) }); } else { $('ul.navigation a:first').click(); } // offset is used to move to *exactly* the right place, since I'm using // padding on my example, I need to subtract the amount of padding to // the offset. Try removing this to get a good idea of the effect var offset = parseInt((horizontal ? $container.css('paddingTop') : $container.css('paddingLeft')) || 0) * -1; var scrollOptions = { target: $scroll, // the element that has the overflow // can be a selector which will be relative to the target items: $panels, navigation: '.navigation a', // selectors are NOT relative to document, i.e. make sure they're unique prev: 'img.left', next: 'img.right', // allow the scroll effect to run both directions axis: 'xy', onAfter: trigger, // our final callback offset: offset, // duration of the sliding effect duration: 500, // easing - can be used with the easing plugin: // http://gsgd.co.uk/sandbox/jquery/easing/ easing: 'swing' }; // apply serialScroll to the slider - we chose this plugin because it // supports// the indexed next and previous scroll along with hooking // in to our navigation. $('#slider').serialScroll(scrollOptions); // now apply localScroll to hook any other arbitrary links to trigger // the effect $.localScroll(scrollOptions); // finally, if the URL has a hash, move the slider in to position, // setting the duration to 1 because I don't want it to scroll in the // very first page load. We don't always need this, but it ensures // the positioning is absolutely spot on when the pages loads. scrollOptions.duration = 1; $.localScroll.hash(scrollOptions); /////////////////////////////////////////////// // autoscroll /////////////////////////////////////////////// // start to automatically cycle the tabs cycleTimer = setInterval(function () { $scroll.trigger('next'); }, 2000); // how many milliseconds, change this to whatever you like // select some trigger elements to stop the auto-cycle var $stopTriggers = $('#slider .navigation').find('a') // tab headers .add('.scroll') // panel itself .add('.stopscroll') // links to the stop class div .add('.navigation') // links to navigation id for tabs .add("a[href^='#']"); // links to a tab // this is the function that will stop the auto-cycle function stopCycle() { // remove the no longer needed stop triggers clearInterval(cycleTimer); // stop the auto-cycle itself $buttons.show(); // show the navigation buttons document.getElementById('stopscroll').style.display='none'; // hide the stop div document.getElementById('startscroll').style.display='block'; // block the start div } // bind stop cycle function to the click event using namespaces $stopTriggers.bind('click.cycle', stopCycle); /////////////////////////////////////////////// // end autoscroll /////////////////////////////////////////////// // edit to start again /////////////////////////////////////////////// // select some trigger elements to stop the auto-cycle var $startTriggers_start = $('#slider .navigation').find('a') // tab headers .add('.startscroll'); // links to the start class div // this is the function that will stop the auto-cycle function startCycle() { // remove the no longer needed stop triggers $buttons.hide(); // show the navigation buttons $scroll.trigger('next'); // directly to the next first cycleTimer = setInterval(function () { // now set timer again $scroll.trigger('next'); }, 5000); // how many milliseconds, change this to whatever you like document.getElementById('stopscroll').style.display='block'; // block the stop div document.getElementById('startscroll').style.display='none'; // hide the start div } // bind stop cycle function to the click event using namespaces $startTriggers_start.bind('click.cycle', startCycle); /////////////////////////////////////////////// // end edit to start /////////////////////////////////////////////// });

  Read the article

• Why wont this sort in Solr work?

  - by Camran

  I need to sort on a date-field type, which name is "mod_date". It works like this in the browser adress-bar: http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+desc But I am using a phpSolr client which sends an URL to Solr, and the url sent is this: fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc // This wont work and is echoed after this in php: $queryString = http_build_query($params, null, $this->_queryStringDelimiter); $queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString); This wont work, I dont know why! Everything else works fine, all right fields are returned. But the sort doesn't work. Any ideas? Thanks BTW: The field "mod_date" contains something like: 2010-03-04T19:37:22.5Z EDIT: First I use PHP to send this to a SolrPhpClient which is another php-file called service.php: require_once('../SolrPhpClient/Apache/Solr/Service.php'); $solr = new Apache_Solr_Service('localhost', 8983, '/solr/'); $results = $solr->search($querystring, $p, $limit, $solr_params); $solr_params is an array which contains the solr-parameters (q, fq, etc). Now, in service.php: $params['version'] = self::SOLR_VERSION; // common parameters in this interface $params['wt'] = self::SOLR_WRITER; $params['json.nl'] = $this->_namedListTreatment; $params['q'] = $query; $params['sort'] = 'mod_date desc'; // HERE IS THE SORT I HAVE PROBLEM WITH $params['start'] = $offset; $params['rows'] = $limit; $queryString = http_build_query($params, null, $this->_queryStringDelimiter); $queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString); if ($method == self::METHOD_GET) { return $this->_sendRawGet($this->_searchUrl . $this->_queryDelimiter . $queryString); } else if ($method == self::METHOD_POST) { return $this->_sendRawPost($this->_searchUrl, $queryString, FALSE, 'application/x-www-form-urlencoded'); } The $results contain the results from Solr... So this is the way I need to get to work (via php). This code below (also on top of this Q) works but thats because I paste it into the adress bar manually, not via the PHPclient. But thats just for debugging, I need to get it to work via the PHPclient: http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+des // Not via phpclient, but works UPDATE (2010-03-08): I have tried Donovans codes (the urls) and they work fine. Now, I have noticed that it is one of the parameters causing the 'SORT' not to work. This parameter is the "wt" parameter. If we take the url from top of this Q, (fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc), and just simply remove the "wt" parameter, then the sort works. BUT the results appear differently, thus making my php code not able to recognize the results I believe. Donovan would know this I think. I am guessing in order for the PHPClient to work, the results must be in a specific structure, which gets messed up as soon as I remove the wt parameter. Donovan, help me please... Here is some background what I use your SolrPhpClient for: I have a classifieds website, which uses MySql. But for the searching I am using Solr to search some indexed fields. Then Solr returns an array of ID:numbers (for all matches of the search criteria). Then I use those ID:numbers to find everything in a MySql db and fetch all other information (example is not searchable information). So simplified: Search - Solr returns all matches in an array of ID:nrs - Id:numbers from Solr are the same as the Id numbers in the MySql db, so I can just make a simple match agains every record with the ID matching the ID from the Solr results array. I don't use Faceting, no boosting, no relevancy or other fancy stuff. I only sort by the latest classified put, and give the option to users to also sort on the cheapest price. Nothing more. Then I use the "fq" parameter to do queries on different fields in Solr depending on category chosen by users (example "cars" in this case which in my language is "Bilar"). I am really stuck with this problem here... Thanks for all help

  Read the article

• New hire expectations... (Am I being unreasonable?)

  - by user295841

  I work for a very small custom software shop. We currently consist me and my boss. My boss is an old FoxPro DOS developer and OOP makes him uncomfortable. He is planning on taking a back seat in the next few years to hopefully enjoy a “partial retirement”. I will be taking over the day to day operations and we are now desperately looking for more help. We tried Monster.com, Dice.com, and others a few years ago when we started our search. We had no success. We have tried outsourcing overseas (total disaster), hiring kids right out of college (mostly a disaster but that’s where I came from), interns (good for them, not so good for us) and hiring laid off “experienced” developers (there was a reason they were laid off). I have heard hiring practices discussed on podcasts, blogs, etc... and have tried a few. The “Fizz Buzz” test was a good one. One kid looked physically ill before he finally gave up. I think my problem is that I have grown so much as a developer since I started here that I now have a high standard. I hear/read very intelligent people podcasts and blogs and I know that there are lots of people out there that can do the job. I don’t want to settle for less than a “good” developer. Perhaps my expectations are unreasonable. I expect any good developer (entry level or experienced) to be billable (at least paying their own wage) in under one month. I expect any good developer to be able to be productive (at least dangerous) in any language or technology with only a few days of research/training. I expect any good developer to be able to take a project from initial customer request to completion with little or no help from others. Am I being unreasonable? What constitutes a valuable developer? What should be expected of an entry level developer? What should be expected of an experienced developer? I realize that everyone is different but there has to be some sort of expectations standard, right? I have been giving the test project below to potential canidates to weed them out. Good idea? Too much? Too little? Please let me know what you think. Thanks. Project ID: T00001 Description: Order Entry System Deadline: 1 Week Scope The scope of this project is to develop a fully function order entry system. Screen/Form design must be user friendly and promote efficient data entry and modification. User experience (Navigation, Screen/Form layouts, Look and Feel…) is at the developer’s discretion. System may be developed using any technologies that conform to the technical and system requirements. Deliverables Complete source code Database setup instructions (Scripts or restorable backup) Application installation instructions (Installer or installation procedure) Any necessary documentation Technical Requirements Server Platform – Windows XP / Windows Server 2003 / SBS Client Platform – Windows XP Web Browser (If applicable) – IE 8 Database – At developer’s discretion (Must be a relational SQL database.) Language – At developer’s discretion All data must be normalized. (+) All data must maintain referential integrity. (++) All data must be indexed for optimal performance. System must handle concurrency. System Requirements Customer Maintenance Customer records must have unique ID. Customer data will include Name, Address, Phone, etc. User must be able to perform all CRUD (Create, Read, Update, and Delete) operations on the Customer table. User must be able to enter a specific Customer ID to edit. User must be able to pull up a sortable/queryable search grid/utility to find a customer to edit. Validation must be performed prior to database commit. Customer record cannot be deleted if the customer has an order in the system. (++) Inventory Maintenance Part records must have unique ID. Part data will include Description, Price, UOM (Unit of Measure), etc. User must be able to perform all CRUD operations on the part table. User must be able to enter a specific Part ID to edit. User must be able to pull up a sortable/queryable search grid/utility to find a part to edit. Validation must be performed prior to database commit. Part record cannot be deleted if the part has been used in an order. (++) Order Entry Order records must have a unique auto-incrementing key (Order Number). Order data must be split into a header/detail structure. (+) Order can contain an infinite number of detail records. Order header data will include Order Number, Customer ID (++), Order Date, Order Status (Open/Closed), etc. Order detail data will include Part Number (++), Quantity, Price, etc. User must be able to perform all CRUD operations on the order tables. User must be able to enter a specific Order Number to edit. User must be able to pull up a sortable/queryable search grid/utility to find an order to edit. User must be able to print an order form from within the order entry form. Validation must be performed prior to database commit. Reports Customer Listing – All Customers in the system. Inventory Listing – All parts in the system. Open Order Listing – All open orders in system. Customer Order Listing – All orders for specific customer. All reports must include sorts and filter functions where applicable. Ex. Customer Listing by range of Customer IDs. Open Order Listing by date range.

  Read the article

• Optimizing python code performance when importing zipped csv to a mongo collection

  - by mark

  I need to import a zipped csv into a mongo collection, but there is a catch - every record contains a timestamp in Pacific Time, which must be converted to the local time corresponding to the (longitude,latitude) pair found in the same record. The code looks like so: def read_csv_zip(path, timezones): with ZipFile(path) as z, z.open(z.namelist()[0]) as input: csv_rows = csv.reader(input) header = csv_rows.next() check,converters = get_aux_stuff(header) for csv_row in csv_rows: if check(csv_row): row = { converter[0]:converter[1](value) for converter, value in zip(converters, csv_row) if allow_field(converter) } ts = row['ts'] lng, lat = row['loc'] found_tz_entry = timezones.find_one(SON({'loc': {'$within': {'$box': [[lng-tz_lookup_radius, lat-tz_lookup_radius],[lng+tz_lookup_radius, lat+tz_lookup_radius]]}}})) if found_tz_entry: tz_name = found_tz_entry['tz'] local_ts = ts.astimezone(timezone(tz_name)).replace(tzinfo=None) row['tz'] = tz_name else: local_ts = (ts.astimezone(utc) + timedelta(hours = int(lng/15))).replace(tzinfo = None) row['local_ts'] = local_ts yield row def insert_documents(collection, source, batch_size): while True: items = list(itertools.islice(source, batch_size)) if len(items) == 0: break; try: collection.insert(items) except: for item in items: try: collection.insert(item) except Exception as exc: print("Failed to insert record {0} - {1}".format(item['_id'], exc)) def main(zip_path): with Connection() as connection: data = connection.mydb.data timezones = connection.timezones.data insert_documents(data, read_csv_zip(zip_path, timezones), 1000) The code proceeds as follows: Every record read from the csv is checked and converted to a dictionary, where some fields may be skipped, some titles be renamed (from those appearing in the csv header), some values may be converted (to datetime, to integers, to floats. etc ...) For each record read from the csv, a lookup is made into the timezones collection to map the record location to the respective time zone. If the mapping is successful - that timezone is used to convert the record timestamp (pacific time) to the respective local timestamp. If no mapping is found - a rough approximation is calculated. The timezones collection is appropriately indexed, of course - calling explain() confirms it. The process is slow. Naturally, having to query the timezones collection for every record kills the performance. I am looking for advises on how to improve it. Thanks. EDIT The timezones collection contains 8176040 records, each containing four values: > db.data.findOne() { "_id" : 3038814, "loc" : [ 1.48333, 42.5 ], "tz" : "Europe/Andorra" } EDIT2 OK, I have compiled a release build of http://toblerity.github.com/rtree/ and configured the rtree package. Then I have created an rtree dat/idx pair of files corresponding to my timezones collection. So, instead of calling collection.find_one I call index.intersection. Surprisingly, not only there is no improvement, but it works even more slowly now! May be rtree could be fine tuned to load the entire dat/idx pair into RAM (704M), but I do not know how to do it. Until then, it is not an alternative. In general, I think the solution should involve parallelization of the task. EDIT3 Profile output when using collection.find_one: >>> p.sort_stats('cumulative').print_stats(10) Tue Apr 10 14:28:39 2012 ImportDataIntoMongo.profile 64549590 function calls (64549180 primitive calls) in 1231.257 seconds Ordered by: cumulative time List reduced from 730 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.012 0.012 1231.257 1231.257 ImportDataIntoMongo.py:1(<module>) 1 0.001 0.001 1230.959 1230.959 ImportDataIntoMongo.py:187(main) 1 853.558 853.558 853.558 853.558 {raw_input} 1 0.598 0.598 370.510 370.510 ImportDataIntoMongo.py:165(insert_documents) 343407 9.965 0.000 359.034 0.001 ImportDataIntoMongo.py:137(read_csv_zip) 343408 2.927 0.000 287.035 0.001 c:\python27\lib\site-packages\pymongo\collection.py:489(find_one) 343408 1.842 0.000 274.803 0.001 c:\python27\lib\site-packages\pymongo\cursor.py:699(next) 343408 2.542 0.000 271.212 0.001 c:\python27\lib\site-packages\pymongo\cursor.py:644(_refresh) 343408 4.512 0.000 253.673 0.001 c:\python27\lib\site-packages\pymongo\cursor.py:605(__send_message) 343408 0.971 0.000 242.078 0.001 c:\python27\lib\site-packages\pymongo\connection.py:871(_send_message_with_response) Profile output when using index.intersection: >>> p.sort_stats('cumulative').print_stats(10) Wed Apr 11 16:21:31 2012 ImportDataIntoMongo.profile 41542960 function calls (41542536 primitive calls) in 2889.164 seconds Ordered by: cumulative time List reduced from 778 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.028 0.028 2889.164 2889.164 ImportDataIntoMongo.py:1(<module>) 1 0.017 0.017 2888.679 2888.679 ImportDataIntoMongo.py:202(main) 1 2365.526 2365.526 2365.526 2365.526 {raw_input} 1 0.766 0.766 502.817 502.817 ImportDataIntoMongo.py:180(insert_documents) 343407 9.147 0.000 491.433 0.001 ImportDataIntoMongo.py:152(read_csv_zip) 343406 0.571 0.000 391.394 0.001 c:\python27\lib\site-packages\rtree-0.7.0-py2.7.egg\rtree\index.py:384(intersection) 343406 379.957 0.001 390.824 0.001 c:\python27\lib\site-packages\rtree-0.7.0-py2.7.egg\rtree\index.py:435(_intersection_obj) 686513 22.616 0.000 38.705 0.000 c:\python27\lib\site-packages\rtree-0.7.0-py2.7.egg\rtree\index.py:451(_get_objects) 343406 6.134 0.000 33.326 0.000 ImportDataIntoMongo.py:162(<dictcomp>) 346 0.396 0.001 30.665 0.089 c:\python27\lib\site-packages\pymongo\collection.py:240(insert) EDIT4 I have parallelized the code, but the results are still not very encouraging. I am convinced it could be done better. See my own answer to this question for details.

  Read the article

• Google App Engine - SiteMap Creation for a social network

  - by spidee

  Hi all. I am creating a social tool - I want to allow search engines to pick up "public" user profiles - like twitter and face-book. I have seen all the protocol info at http://www.sitemaps.org and i understand this and how to build such a file - along with an index if i exceed the 50K limit. Where i am struggling is the concept of how i make this run. The site map for my general site pages is simple i can use a tool to create the file - or a script - host the file - submit the file and done. What i then need is a script that will create the site-maps of user profiles. I assume this would be something like: <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.socialsite.com/profile/spidee</loc> <lastmod>2010-5-12</lastmod> <changefreq>???</changefreq> <priority>???</priority> </url> <url> <loc>http://www.socialsite.com/profile/webbsterisback</loc> <lastmod>2010-5-12</lastmod> <changefreq>???</changefreq> <priority>???</priority> </url> </urlset> Ive added some ??? as i don't know how i should set these settings for my profiles based on the following:- When a new profile is created it must be added to a site-map. If the profile is changed or if "certain" properties are changed - then i don't know if i update the entry in the map - or do something else? (updating would be a nightmare!) Some users may change their profile. In terms of relevance to the search engine the only way a google or yahoo search will find the users (for my requirement) profile would be for example by means of [user name] and [location] so once the entry for the profile has been added to the map file the only reason to have the search-bot re-index the profile would be if the user changed their user-name - which they cant. or their location - and or set their settings so that their profile would be "hidden" from search engines. I assume my map creation will need to be dynamic. From what i have said above i would imagine that creating a new profile and possible editing certain properties could mark it as needing adding/updating in the sitemap. Assuming i will have millions of profiles added/being edited how can i manage this in a sensible manner. i know i need a script that can append urls as each profile is created i know the script will prob be a TASK - running at a set freq - perhaps the profiles have a property like "indexed" and the TASK sets them to "true" when the profiles are added to the map. I dont see the best way to store the map - do i store it in the datastore i.e; model=sitemaps properties key_name=sitemap_xml_1 (and for my map sitemap_index_xml) mapxml=blobstore (the raw xml map or ror map) full=boolean (set true when url count is 50) # might need this as a shard will tell us To make this work my thoughts are m cache the current site map structure as "sitemap_xml" keep a shard of url count when my task executes 1. build the xml structure for say the first 100 urls marked "index==false" (how many could u run at a time?) 2. test if the current mcache sitemap is full (shardcounter+10050K) 3.a if the map is near full create a new map entry in models "sitemap_xml_2" - update the map_index file (also stored in my model as "sitemap_index" start a new shard - or reset.2 3.b if the map is not full grab it from mcache 4.append the 100 url xml structure 5.save / m cache the map I can now add a handler using a url map/route like /sitemaps/* Get my * as map name and serve the maps from the blobstore/mache on the fly. Now my question is does this work - is this the right way or a good way to start? Will this handle the situation of making sure the search bots update when a user changes their profile - possibly by setting the change freq correctly? - Do i need a more advance system :( ? or have i re-invented the wheel! I hope this is all clear and make some form of sense :-)

  Read the article

• php foreach looping twice

  - by Jack

  Hi, I am trying to loop through some data from my database but it is outputting it twice. $fields = 'field1, field2, field3, field4'; $idFields = 'id_field1, id_field2, id_field3, id_field4'; $tables = 'table1, table2, table3, table4'; $table = explode(', ', $tables); $field = explode(', ', $fields); $id = explode(', ', $idFields); $str = 'Egg'; $i=1; while ($i<4) { $f = $field[$i]; $idd = $id[$i]; $sql = $writeConn->select()->from($table[$i], array($f, $idd))->where($f . " LIKE ?", '%' . $str . '%'); $string = '<a title="' . $str . '" href="' . $currentProductUrl . '">' . $str . '</a>'; $result = $writeConn->fetchAssoc($sql); foreach ($result as $row) { echo 'Success! Found ' . $str . ' in ' . $f . '. ID: ' . $row[$idd] . '.<br>'; } $i++; } Outputting: Success! Found Egg in field3. ID: 5. Success! Found Egg in field3. ID: 5. Could someone please explain why it is looping through both the indexed and associative values? UPDATE I did some more playing around and tried the following. $fields = 'field1, field2, field3, field4'; $idFields = 'id_field1, id_field2, id_field3, id_field4'; $tables = 'table1, table2, table3, table4'; $table = explode(', ', $tables); $field = explode(', ', $fields); $id = explode(', ', $idFields); $str = 'Egg'; $i=1; while ($i<4) { $f = $field[$i]; $idd = $id[$i]; $sql = $writeConn->select()->from($table[$i], array($f, $idd))->where($f . " LIKE ?", '%' . $str . '%'); $string = '<a title="' . $str . '" href="' . $currentProductUrl . '">' . $str . '</a>'; $sth = $writeConn->prepare($sql); $sth->execute(); $result = $sth->fetch(PDO::FETCH_ASSOC); foreach ($result as $row) { echo 'Success! Found ' . $str . ' in ' . $f . '. ID: ' . $row[$idd] . '.<br>'; } $i++; } The interesting thing is that this outputs the below: Success! Found Egg in field3. ID: E. Success! Found Egg in field3. ID: E. Success! Found Egg in field3. ID: 5. Success! Found Egg in field3. ID: 5. Success! Found Egg in field3. ID: E. Success! Found Egg in field3. ID: E. Success! Found Egg in field3. ID: 5. Success! Found Egg in field3. ID: 5. I have also tried adding $i to the output and this outputs 2 as expected. If I change fetch(PDO::FETCH_BOTH) to fetch(PDO::FETCH_ASSOC) the output is as follows: Success! Found Egg in field3. ID: E. Success! Found Egg in field3. ID: E. Success! Found Egg in field3. ID: 5. Success! Found Egg in field3. ID: 5. This has been bugging me for too long, so if anyone could help I would be very appreciative!

  Read the article

• iPhone SDK Tableview Datasource singleton error

  - by mrburns05

  I basically followed apple "TheElements" sample and changed "PeriodicElements" .h & .m to my own "SortedItems" .h & .m During compile I get this error: "Undefined symbols: "_OBJC_CLASS_$_SortedItems", referenced from: __objc_classrefs__DATA@0 in SortedByNameTableDataSource.o ld: symbol(s) not found collect2: ld returned 1 exit status " here is my SortedItems.m file #import "SortedItems.h" #import "item.h" #import "MyAppDelegate.h" @interface SortedItems(mymethods) // these are private methods that outside classes need not use - (void)presortItemsByPhysicalState; - (void)presortItemInitialLetterIndexes; - (void)presortItemNamesForInitialLetter:(NSString *)aKey; - (void)presortItemsWithPhysicalState:(NSString *)state; - (NSArray *)presortItemsByNumber; - (NSArray *)presortItemsBySymbol; - (void)setupItemsArray; @end @implementation SortedItems @synthesize statesDictionary; @synthesize itemsDictionary; @synthesize nameIndexesDictionary; @synthesize itemNameIndexArray; @synthesize itemsSortedByNumber; @synthesize itemsSortedBySymbol; @synthesize itemPhysicalStatesArray; static SortedItems *sharedSortedItemsInstance = nil; + (SortedItems*)sharedSortedItems { @synchronized(self) { if (sharedSortedItemsInstance == nil) { [[self alloc] init]; // assignment not done here } } return sharedSortedItemsInstance; // note: Xcode (3.2) static analyzer will report this singleton as a false positive // '(Potential leak of an object allocated') } + (id)allocWithZone:(NSZone *)zone { @synchronized(self) { if (sharedSortedItemsInstance == nil) { sharedSortedItemsInstance = [super allocWithZone:zone]; return sharedSortedItemsInstance; // assignment and return on first allocation } } return nil; //on subsequent allocation attempts return nil } - (id)copyWithZone:(NSZone *)zone { return self; } - (id)retain { return self; } - (unsigned)retainCount { return UINT_MAX; //denotes an object that cannot be released } - (void)release { //do nothing } - (id)autorelease { return self; } // setup the data collection - init { if (self = [super init]) { [self setupItemsArray]; } return self; } - (void)setupItemsArray { NSDictionary *eachItem; // create dictionaries that contain the arrays of Item data indexed by // name self.itemsDictionary = [NSMutableDictionary dictionary]; // physical state self.statesDictionary = [NSMutableDictionary dictionary]; // unique first characters (for the Name index table) self.nameIndexesDictionary = [NSMutableDictionary dictionary]; // create empty array entries in the states Dictionary or each physical state [statesDictionary setObject:[NSMutableArray array] forKey:@"Solid"]; [statesDictionary setObject:[NSMutableArray array] forKey:@"Liquid"]; [statesDictionary setObject:[NSMutableArray array] forKey:@"Gas"]; [statesDictionary setObject:[NSMutableArray array] forKey:@"Artificial"]; MyAppDelegate *ad = (MyAppDelegate *)[[UIApplication sharedApplication]delegate]; NSMutableArray *rawItemsArray = [[NSMutableArray alloc] init]; [rawItemsArray addObjectsFromArray:ad.items]; // iterate over the values in the raw Items dictionary for (eachItem in rawItemsArray) { // create an atomic Item instance for each Item *anItem = [[Item alloc] initWithDictionary:eachItem]; // store that item in the Items dictionary with the name as the key [itemsDictionary setObject:anItem forKey:anItem.title]; // add that Item to the appropriate array in the physical state dictionary [[statesDictionary objectForKey:anItem.acct] addObject:anItem]; // get the Item's initial letter NSString *firstLetter = [anItem.title substringToIndex:1]; NSMutableArray *existingArray; // if an array already exists in the name index dictionary // simply add the Item to it, otherwise create an array // and add it to the name index dictionary with the letter as the key if (existingArray = [nameIndexesDictionary valueForKey:firstLetter]) { [existingArray addObject:anItem]; } else { NSMutableArray *tempArray = [NSMutableArray array]; [nameIndexesDictionary setObject:tempArray forKey:firstLetter]; [tempArray addObject:anItem]; } // release the Item, it is held by the various collections [anItem release]; } // release the raw Item data [rawItemsArray release]; // create the dictionary containing the possible Item states // and presort the states data self.itemPhysicalStatesArray = [NSArray arrayWithObjects:@"something",@"somethingElse",@"whatever",@"stuff",nil]; [self presortItemsByPhysicalState]; // presort the dictionaries now // this could be done the first time they are requested instead [self presortItemInitialLetterIndexes]; self.itemsSortedByNumber = [self presortItemsByNumber]; self.itemsSortedBySymbol = [self presortItemsBySymbol]; } // return the array of Items for the requested physical state - (NSArray *)itemsWithPhysicalState:(NSString*)aState { return [statesDictionary objectForKey:aState]; } // presort each of the arrays for the physical states - (void)presortItemsByPhysicalState { for (NSString *stateKey in itemPhysicalStatesArray) { [self presortItemsWithPhysicalState:stateKey]; } } - (void)presortItemsWithPhysicalState:(NSString *)state { NSSortDescriptor *nameDescriptor = [[NSSortDescriptor alloc] initWithKey:@"title" ascending:YES selector:@selector(localizedCaseInsensitiveCompare:)] ; NSArray *descriptors = [NSArray arrayWithObject:nameDescriptor]; [[statesDictionary objectForKey:state] sortUsingDescriptors:descriptors]; [nameDescriptor release]; } // return an array of Items for an initial letter (ie A, B, C, ...) - (NSArray *)itemsWithInitialLetter:(NSString*)aKey { return [nameIndexesDictionary objectForKey:aKey]; } // presort the name index arrays so the items are in the correct order - (void)presortItemsInitialLetterIndexes { self.itemNameIndexArray = [[nameIndexesDictionary allKeys] sortedArrayUsingSelector:@selector(localizedCaseInsensitiveCompare:)]; for (NSString *eachNameIndex in itemNameIndexArray) { [self presortItemNamesForInitialLetter:eachNameIndex]; } } - (void)presortItemNamesForInitialLetter:(NSString *)aKey { NSSortDescriptor *nameDescriptor = [[NSSortDescriptor alloc] initWithKey:@"title" ascending:YES selector:@selector(localizedCaseInsensitiveCompare:)] ; NSArray *descriptors = [NSArray arrayWithObject:nameDescriptor]; [[nameIndexesDictionary objectForKey:aKey] sortUsingDescriptors:descriptors]; [nameDescriptor release]; } // presort the ItemsSortedByNumber array - (NSArray *)presortItemsByNumber { NSSortDescriptor *nameDescriptor = [[NSSortDescriptor alloc] initWithKey:@"acct" ascending:YES selector:@selector(compare:)] ; NSArray *descriptors = [NSArray arrayWithObject:nameDescriptor]; NSArray *sortedItems = [[itemsDictionary allValues] sortedArrayUsingDescriptors:descriptors]; [nameDescriptor release]; return sortedItems; } // presort the itemsSortedBySymbol array - (NSArray *)presortItemsBySymbol { NSSortDescriptor *symbolDescriptor = [[NSSortDescriptor alloc] initWithKey:@"title" ascending:YES selector:@selector(localizedCaseInsensitiveCompare:)] ; NSArray *descriptors = [NSArray arrayWithObject:symbolDescriptor]; NSArray *sortedItems = [[itemsDictionary allValues] sortedArrayUsingDescriptors:descriptors]; [symbolDescriptor release]; return sortedItems; } @end I followed the sample exactly - don't know where I went wrong. Here is my "SortedByNameTableDataSource.m" #import "SortedByNameTableDataSource.h" #import "SortedItems.h" #import "Item.h" #import "ItemCell.h" #import "GradientView.h" #import "UIColor-Expanded.h" #import "MyAppDelegate.h" @implementation SortedByNameTableDataSource - (NSString *)title { return @"Title"; } - (UITableViewStyle)tableViewStyle { return UITableViewStylePlain; }; // return the atomic element at the index - (Item *)itemForIndexPath:(NSIndexPath *)indexPath { return [[[SortedItems sharedSortedItems] itemsWithInitialLetter:[[[SortedItems sharedSortedItems] itemNameIndexArray] objectAtIndex:indexPath.section]] objectAtIndex:indexPath.row]; } // UITableViewDataSource methods - (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath { static NSString *MyIdentifier = @"ItemCell"; ItemCell *itemCell = (ItemCell *)[tableView dequeueReusableCellWithIdentifier:MyIdentifier]; if (itemCell == nil) { itemCell = [[[ItemCell alloc] initWithFrame:CGRectZero reuseIdentifier:MyIdentifier] autorelease]; itemCell = CGRectMake(0.0, 0.0, 320.0, ROW_HEIGHT); itemCell.backgroundView = [[[GradientView alloc] init] autorelease]; } itemCell.todo = [self itemForIndexPath:indexPath]; return itemCell; } - (NSInteger)numberOfSectionsInTableView:(UITableView *)tableView { // this table has multiple sections. One for each unique character that an element begins with // [A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,U,V,X,Y,Z] // return the count of that array return [[[SortedItems sharedSortedItems] itemNameIndexArray] count]; } - (NSArray *)sectionIndexTitlesForTableView:(UITableView *)tableView { // returns the array of section titles. There is one entry for each unique character that an element begins with // [A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,U,V,X,Y,Z] return [[SortedItems sharedSortedItems] itemNameIndexArray]; } - (NSInteger)tableView:(UITableView *)tableView sectionForSectionIndexTitle:(NSString *)title atIndex:(NSInteger)index { return index; } - (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section { // the section represents the initial letter of the element // return that letter NSString *initialLetter = [[[SortedItems sharedSortedItems] itemNameIndexArray] objectAtIndex:section]; // get the array of elements that begin with that letter NSArray *itemsWithInitialLetter = [[SortedItems sharedSortedItems] itemsWithInitialLetter:initialLetter]; // return the count return [itemsWithInitialLetter count]; } - (NSString *)tableView:(UITableView *)tableView titleForHeaderInSection:(NSInteger)section { // this table has multiple sections. One for each unique character that an element begins with // [A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,U,V,X,Y,Z] // return the letter that represents the requested section // this is actually a delegate method, but we forward the request to the datasource in the view controller return [[[SortedItems sharedSortedItems] itemNameIndexArray] objectAtIndex:section]; } @end

  Read the article

• High Load mysql on Debian server stops every day. Why?

  - by Oleg Abrazhaev

  I have Debian server with 32 gb memory. And there is apache2, memcached and nginx on this server. Memory load always on maximum. Only 500m free. Most memory leak do MySql. Apache only 70 clients configured, other services small memory usage. When mysql use all memory it stops. And nothing works, need mysql reboot. Mysql configured use maximum 24 gb memory. I have hight weight InnoDB bases. (400000 rows, 30 gb). And on server multithread daemon, that makes many inserts in this tables, thats why InnoDB. There is my mysql config. [mysqld] # # * Basic Settings # default-time-zone = "+04:00" user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp language = /usr/share/mysql/english skip-external-locking default-time-zone='Europe/Moscow' # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. # # * Fine Tuning # #low_priority_updates = 1 concurrent_insert = ALWAYS wait_timeout = 600 interactive_timeout = 600 #normal key_buffer_size = 2024M #key_buffer_size = 1512M #70% hot cache key_cache_division_limit= 70 #16-32 max_allowed_packet = 32M #1-16M thread_stack = 8M #40-50 thread_cache_size = 50 #orderby groupby sort sort_buffer_size = 64M #same myisam_sort_buffer_size = 400M #temp table creates when group_by tmp_table_size = 3000M #tables in memory max_heap_table_size = 3000M #on disk open_files_limit = 10000 table_cache = 10000 join_buffer_size = 5M # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched myisam-recover = BACKUP #myisam_use_mmap = 1 max_connections = 200 thread_concurrency = 8 # # * Query Cache Configuration # #more ignored query_cache_limit = 50M query_cache_size = 210M #on query cache query_cache_type = 1 # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. #log = /var/log/mysql/mysql.log # # Error logging goes to syslog. This is a Debian improvement :) # # Here you can see queries with especially long duration log_slow_queries = /var/log/mysql/mysql-slow.log long_query_time = 1 log-queries-not-using-indexes # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #log_bin = /var/log/mysql/mysql-bin.log server-id = 1 log-bin = /var/lib/mysql/mysql-bin #replicate-do-db = gate log-bin-index = /var/lib/mysql/mysql-bin.index log-error = /var/lib/mysql/mysql-bin.err relay-log = /var/lib/mysql/relay-bin relay-log-info-file = /var/lib/mysql/relay-bin.info relay-log-index = /var/lib/mysql/relay-bin.index binlog_do_db = 24avia expire_logs_days = 10 max_binlog_size = 100M read_buffer_size = 4024288 innodb_buffer_pool_size = 5000M innodb_flush_log_at_trx_commit = 2 innodb_thread_concurrency = 8 table_definition_cache = 2000 group_concat_max_len = 16M #binlog_do_db = gate #binlog_ignore_db = include_database_name # # * BerkeleyDB # # Using BerkeleyDB is now discouraged as its support will cease in 5.1.12. #skip-bdb # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! # You might want to disable InnoDB to shrink the mysqld process by circa 100MB. #skip-innodb # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates I recommend the OpenSSL GUI "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pem [mysqldump] quick quote-names max_allowed_packet = 500M [mysql] #no-auto-rehash # faster start of mysql but no tab completition [isamchk] key_buffer = 32M key_buffer_size = 512M # # * NDB Cluster # # See /usr/share/doc/mysql-server-*/README.Debian for more information. # # The following configuration is read by the NDB Data Nodes (ndbd processes) # not from the NDB Management Nodes (ndb_mgmd processes). # # [MYSQL_CLUSTER] # ndb-connectstring=127.0.0.1 # # * IMPORTANT: Additional settings that can override those from this file! # The files must end with '.cnf', otherwise they'll be ignored. # !includedir /etc/mysql/conf.d/ Please, help me make it stable. Memory used /etc/mysql # free total used free shared buffers cached Mem: 32930800 32766424 164376 0 139208 23829196 -/+ buffers/cache: 8798020 24132780 Swap: 33553328 44660 33508668 Maybe my problem not in memory, but MySQL stops every day. As you can see, cache memory free 24 gb. Thank to Michael Hampton? for correction. Load overage on server 3.5. Maybe hdd or another problem? Maybe my config not optimal for 30gb InnoDB ? I'm already try mysqltuner and tunung-primer.sh , but they marked all green. Mysqltuner output mysqltuner >> MySQLTuner 1.0.1 - Major Hayden <[email protected]> >> Bug reports, feature requests, and downloads at http://mysqltuner.com/ >> Run with '--help' for additional options and output filtering -------- General Statistics -------------------------------------------------- [--] Skipped version check for MySQLTuner script [OK] Currently running supported MySQL version 5.5.24-9-log [OK] Operating on 64-bit architecture -------- Storage Engine Statistics ------------------------------------------- [--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster [--] Data in MyISAM tables: 112G (Tables: 1528) [--] Data in InnoDB tables: 39G (Tables: 340) [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17) [!!] Total fragmented tables: 344 -------- Performance Metrics ------------------------------------------------- [--] Up for: 8h 18m 33s (14M q [478.333 qps], 259K conn, TX: 9B, RX: 5B) [--] Reads / Writes: 84% / 16% [--] Total buffers: 10.5G global + 81.1M per thread (200 max threads) [OK] Maximum possible memory usage: 26.3G (83% of installed RAM) [OK] Slow queries: 1% (259K/14M) [!!] Highest connection usage: 100% (201/200) [OK] Key buffer size / total MyISAM indexes: 1.5G/5.6G [OK] Key buffer hit rate: 100.0% (6B cached / 1M reads) [OK] Query cache efficiency: 74.3% (8M cached / 11M selects) [OK] Query cache prunes per day: 0 [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 247K sorts) [!!] Joins performed without indexes: 106025 [!!] Temporary tables created on disk: 49% (351K on disk / 715K total) [OK] Thread cache hit rate: 99% (249 created / 259K connections) [!!] Table cache hit rate: 15% (2K open / 13K opened) [OK] Open file limit used: 15% (3K/20K) [OK] Table locks acquired immediately: 99% (4M immediate / 4M locks) [!!] InnoDB data size / buffer pool: 39.4G/5.9G -------- Recommendations ----------------------------------------------------- General recommendations: Run OPTIMIZE TABLE to defragment tables for better performance MySQL started within last 24 hours - recommendations may be inaccurate Reduce or eliminate persistent connections to reduce connection usage Adjust your join queries to always utilize indexes Temporary table size is already large - reduce result set size Reduce your SELECT DISTINCT queries without LIMIT clauses Increase table_cache gradually to avoid file descriptor limits Variables to adjust: max_connections (> 200) wait_timeout (< 600) interactive_timeout (< 600) join_buffer_size (> 5.0M, or always use indexes with joins) table_cache (> 10000) innodb_buffer_pool_size (>= 39G) Mysql primer output -- MYSQL PERFORMANCE TUNING PRIMER -- - By: Matthew Montgomery - MySQL Version 5.5.24-9-log x86_64 Uptime = 0 days 8 hrs 20 min 50 sec Avg. qps = 478 Total Questions = 14369568 Threads Connected = 16 Warning: Server has not been running for at least 48hrs. It may not be safe to use these recommendations To find out more information on how each of these runtime variables effects performance visit: http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html Visit http://www.mysql.com/products/enterprise/advisors.html for info about MySQL's Enterprise Monitoring and Advisory Service SLOW QUERIES The slow query log is enabled. Current long_query_time = 1.000000 sec. You have 260626 out of 14369701 that take longer than 1.000000 sec. to complete Your long_query_time seems to be fine BINARY UPDATE LOG The binary update log is enabled Binlog sync is not enabled, you could loose binlog records during a server crash WORKER THREADS Current thread_cache_size = 50 Current threads_cached = 45 Current threads_per_sec = 0 Historic threads_per_sec = 0 Your thread_cache_size is fine MAX CONNECTIONS Current max_connections = 200 Current threads_connected = 11 Historic max_used_connections = 201 The number of used connections is 100% of the configured maximum. You should raise max_connections INNODB STATUS Current InnoDB index space = 214 M Current InnoDB data space = 39.40 G Current InnoDB buffer pool free = 0 % Current innodb_buffer_pool_size = 5.85 G Depending on how much space your innodb indexes take up it may be safe to increase this value to up to 2 / 3 of total system memory MEMORY USAGE Max Memory Ever Allocated : 23.46 G Configured Max Per-thread Buffers : 15.84 G Configured Max Global Buffers : 7.54 G Configured Max Memory Limit : 23.39 G Physical Memory : 31.40 G Max memory limit seem to be within acceptable norms KEY BUFFER Current MyISAM index space = 5.61 G Current key_buffer_size = 1.47 G Key cache miss rate is 1 : 5578 Key buffer free ratio = 77 % Your key_buffer_size seems to be fine QUERY CACHE Query cache is enabled Current query_cache_size = 200 M Current query_cache_used = 101 M Current query_cache_limit = 50 M Current Query cache Memory fill ratio = 50.59 % Current query_cache_min_res_unit = 4 K MySQL won't cache query results that are larger than query_cache_limit in size SORT OPERATIONS Current sort_buffer_size = 64 M Current read_rnd_buffer_size = 256 K Sort buffer seems to be fine JOINS Current join_buffer_size = 5.00 M You have had 106606 queries where a join could not use an index properly You have had 8 joins without keys that check for key usage after each row join_buffer_size >= 4 M This is not advised You should enable "log-queries-not-using-indexes" Then look for non indexed joins in the slow query log. OPEN FILES LIMIT Current open_files_limit = 20210 files The open_files_limit should typically be set to at least 2x-3x that of table_cache if you have heavy MyISAM usage. Your open_files_limit value seems to be fine TABLE CACHE Current table_open_cache = 10000 tables Current table_definition_cache = 2000 tables You have a total of 1910 tables You have 2151 open tables. The table_cache value seems to be fine TEMP TABLES Current max_heap_table_size = 2.92 G Current tmp_table_size = 2.92 G Of 366426 temp tables, 49% were created on disk Perhaps you should increase your tmp_table_size and/or max_heap_table_size to reduce the number of disk-based temporary tables Note! BLOB and TEXT columns are not allow in memory tables. If you are using these columns raising these values might not impact your ratio of on disk temp tables. TABLE SCANS Current read_buffer_size = 3 M Current table scan ratio = 2846 : 1 read_buffer_size seems to be fine TABLE LOCKING Current Lock Wait ratio = 1 : 185 You may benefit from selective use of InnoDB. If you have long running SELECT's against MyISAM tables and perform frequent updates consider setting 'low_priority_updates=1'

  Read the article

• IntellIJ editor doesn't appear

  - by neveu

  I updated my IntellIJ install to the latest version (11.1.4) and now the Editor window doesn't appear. Double-clicking on the file, jump-to-source, nothing happens. No error message, it just doesn't appear. If I double-click on an xml layout file the preview window works, but no Editor window. Have installed and reinstalled; went back to an earlier version and it doesn't work there either. I'm at a loss. Any ideas? Update: Editor works if I create a new project. Update 2: idea.log file includes this (I don't know what ins.android.sdk.AndroidSdkData is): 2012-11-04 20:40:52,481 [ 2677] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/macros.xml file is null 2012-11-04 20:40:52,481 [ 2677] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/macros.xml 2012-11-04 20:40:52,482 [ 2678] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/quicklists.xml file is null 2012-11-04 20:40:52,482 [ 2678] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/quicklists.xml 2012-11-04 20:40:52,564 [ 2760] INFO - pl.stores.ApplicationStoreImpl - 76 application components initialized in 1285 ms 2012-11-04 20:40:52,575 [ 2771] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/customization.xml file is null 2012-11-04 20:40:52,575 [ 2771] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/customization.xml 2012-11-04 20:40:52,674 [ 2870] INFO - ij.openapi.wm.impl.IdeRootPane - App initialization took 3385 ms 2012-11-04 20:40:53,136 [ 3332] INFO - TestNG Runner - Create TestNG Template Configuration 2012-11-04 20:40:53,138 [ 3334] INFO - TestNG Runner - Create TestNG Template Configuration 2012-11-04 20:40:53,253 [ 3449] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/dynamic.xml file is null 2012-11-04 20:40:53,253 [ 3449] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/dynamic.xml 2012-11-04 20:40:53,280 [ 3476] INFO - api.vfs.impl.local.FileWatcher - 1 paths checked, 0 mapped, 202 mks 2012-11-04 20:40:53,366 [ 3562] INFO - ellij.project.impl.ProjectImpl - 137 project components initialized in 403 ms 2012-11-04 20:40:53,563 [ 3759] INFO - .module.impl.ModuleManagerImpl - 4 modules loaded in 197 ms 2012-11-04 20:40:53,625 [ 3821] INFO - api.vfs.impl.local.FileWatcher - 6 paths checked, 0 mapped, 150 mks 2012-11-04 20:40:54,187 [ 4383] INFO - .roots.impl.DirectoryIndexImpl - Directory index initialized in 271 ms, indexed 1611 directories 2012-11-04 20:40:54,207 [ 4403] INFO - pl.PushedFilePropertiesUpdater - File properties pushed in 18 ms 2012-11-04 20:40:54,237 [ 4433] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/plainTextFiles.xml file is null 2012-11-04 20:40:54,237 [ 4433] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/plainTextFiles.xml 2012-11-04 20:40:54,246 [ 4442] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/gant_config.xml file is null 2012-11-04 20:40:54,246 [ 4442] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/gant_config.xml 2012-11-04 20:40:54,253 [ 4449] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/gradle.xml file is null 2012-11-04 20:40:54,253 [ 4449] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/gradle.xml 2012-11-04 20:40:55,855 [ 6051] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/IntelliLang.xml file is null 2012-11-04 20:40:55,855 [ 6051] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/IntelliLang.xml 2012-11-04 20:40:56,995 [ 7191] INFO - leEditor.impl.EditorsSplitters - splitter 2012-11-04 20:40:56,996 [ 7192] INFO - leEditor.impl.EditorsSplitters - splitter 2012-11-04 20:40:57,233 [ 7429] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/codeStyleSettings.xml file is null 2012-11-04 20:40:57,233 [ 7429] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/codeStyleSettings.xml 2012-11-04 20:40:57,234 [ 7430] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/projectCodeStyle.xml file is null 2012-11-04 20:40:57,234 [ 7430] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/projectCodeStyle.xml 2012-11-04 20:40:58,145 [ 8341] INFO - indexing.UnindexedFilesUpdater - Indexable files iterated in 3911 ms 2012-11-04 20:40:58,146 [ 8342] INFO - indexing.UnindexedFilesUpdater - Unindexed files update started: 0 files to update 2012-11-04 20:40:58,146 [ 8342] INFO - indexing.UnindexedFilesUpdater - Unindexed files update done in 0 ms 2012-11-04 20:40:58,362 [ 8558] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/fileColors.xml file is null 2012-11-04 20:40:58,362 [ 8558] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $PROJECT_CONFIG_DIR$/fileColors.xml 2012-11-04 20:41:00,420 [ 10616] INFO - ins.android.sdk.AndroidSdkData - For input string: "20.0.1" java.lang.NumberFormatException: For input string: "20.0.1" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:458) at java.lang.Integer.parseInt(Integer.java:499) at org.jetbrains.android.sdk.AndroidSdkData.parsePackageRevision(AndroidSdkData.java:86) at org.jetbrains.android.sdk.AndroidSdkData.<init>(AndroidSdkData.java:73) at org.jetbrains.android.sdk.AndroidSdkData.parse(AndroidSdkData.java:167) at org.jetbrains.android.sdk.AndroidPlatform.parse(AndroidPlatform.java:83) at org.jetbrains.android.sdk.AndroidSdkAdditionalData.getAndroidPlatform(AndroidSdkAdditionalData.java:119) at org.jetbrains.android.facet.AndroidFacet.addResourceFolderToSdkRootsIfNecessary(AndroidFacet.java:532) at org.jetbrains.android.facet.AndroidFacet.access$500(AndroidFacet.java:103) at org.jetbrains.android.facet.AndroidFacet$3.run(AndroidFacet.java:440) at com.intellij.ide.startup.impl.StartupManagerImpl$6.run(StartupManagerImpl.java:230) at com.intellij.ide.startup.impl.StartupManagerImpl.runActivities(StartupManagerImpl.java:203) at com.intellij.ide.startup.impl.StartupManagerImpl.access$100(StartupManagerImpl.java:41) at com.intellij.ide.startup.impl.StartupManagerImpl$4.run(StartupManagerImpl.java:170) at com.intellij.openapi.project.DumbServiceImpl.updateFinished(DumbServiceImpl.java:213) at com.intellij.openapi.project.DumbServiceImpl.access$1000(DumbServiceImpl.java:51) at com.intellij.openapi.project.DumbServiceImpl$IndexUpdateRunnable$1$3.run(DumbServiceImpl.java:363) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:702) at java.awt.EventQueue.access$400(EventQueue.java:82) at java.awt.EventQueue$2.run(EventQueue.java:663) at java.awt.EventQueue$2.run(EventQueue.java:661) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87) at java.awt.EventQueue.dispatchEvent(EventQueue.java:672) at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:699) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:538) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:420) at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:378) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188) at java.awt.EventDispatchThread.run(EventDispatchThread.java:122) 2012-11-04 20:41:00,459 [ 10655] INFO - ins.android.sdk.AndroidSdkData - For input string: "20.0.1" java.lang.NumberFormatException: For input string: "20.0.1" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:458) at java.lang.Integer.parseInt(Integer.java:499) at org.jetbrains.android.sdk.AndroidSdkData.parsePackageRevision(AndroidSdkData.java:86) at org.jetbrains.android.sdk.AndroidSdkData.<init>(AndroidSdkData.java:73) at org.jetbrains.android.sdk.AndroidSdkData.parse(AndroidSdkData.java:167) at org.jetbrains.android.sdk.AndroidPlatform.parse(AndroidPlatform.java:83) at org.jetbrains.android.sdk.AndroidSdkAdditionalData.getAndroidPlatform(AndroidSdkAdditionalData.java:119) at org.jetbrains.android.facet.AndroidFacet.addResourceFolderToSdkRootsIfNecessary(AndroidFacet.java:532) at org.jetbrains.android.facet.AndroidFacet.access$500(AndroidFacet.java:103) at org.jetbrains.android.facet.AndroidFacet$3.run(AndroidFacet.java:440) at com.intellij.ide.startup.impl.StartupManagerImpl$6.run(StartupManagerImpl.java:230) at com.intellij.ide.startup.impl.StartupManagerImpl.runActivities(StartupManagerImpl.java:203) at com.intellij.ide.startup.impl.StartupManagerImpl.access$100(StartupManagerImpl.java:41) at com.intellij.ide.startup.impl.StartupManagerImpl$4.run(StartupManagerImpl.java:170) at com.intellij.openapi.project.DumbServiceImpl.updateFinished(DumbServiceImpl.java:213) at com.intellij.openapi.project.DumbServiceImpl.access$1000(DumbServiceImpl.java:51) at com.intellij.openapi.project.DumbServiceImpl$IndexUpdateRunnable$1$3.run(DumbServiceImpl.java:363) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:702) at java.awt.EventQueue.access$400(EventQueue.java:82) at java.awt.EventQueue$2.run(EventQueue.java:663) at java.awt.EventQueue$2.run(EventQueue.java:661) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87) at java.awt.EventQueue.dispatchEvent(EventQueue.java:672) at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:699) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:538) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:420) at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:378) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188) at java.awt.EventDispatchThread.run(EventDispatchThread.java:122) 2012-11-04 20:41:01,305 [ 11501] INFO - tor.impl.FileEditorManagerImpl - Project opening took 8374 ms 2012-11-04 20:41:01,719 [ 11915] INFO - dom.attrs.AttributeDefinitions - Found tag with unknown parent: AndroidManifest.AndroidManifestCompatibleScreens 2012-11-04 20:41:07,522 [ 17718] INFO - roid.compiler.tools.AndroidApt - [/Users/neveu/Dev/android-sdk-macosx/platform-tools/aapt] [package] [-m] [--non-constant-id] [-J] [/private/var/folders/xb/hg6cdxt51rs8lylmmjw0fk8m0000gp/T/android_apt_autogeneration6157451500950136901tmp] [-M] [/Users/neveu/Dev/magic_android/3rdParty/facebook/AndroidManifest.xml] [-S] [/Users/neveu/Dev/magic_android/3rdParty/facebook/res] [-I] [/Users/neveu/Dev/android-sdk-macosx/platforms/android-14/android.jar] 2012-11-04 20:41:08,706 [ 18902] INFO - roid.compiler.tools.AndroidApt - [/Users/neveu/Dev/android-sdk-macosx/platform-tools/aapt] [package] [-m] [-J] [/private/var/folders/xb/hg6cdxt51rs8lylmmjw0fk8m0000gp/T/android_apt_autogeneration3143184519400737414tmp] [-M] [/Users/neveu/Dev/magic_android/AndroidManifest.xml] [-S] [/Users/neveu/Dev/magic_android/res] [-I] [/Users/neveu/Dev/android-sdk-macosx/platforms/android-15/android.jar] 2012-11-04 20:41:08,763 [ 18959] INFO - roid.compiler.tools.AndroidIdl - [/Users/neveu/Dev/android-sdk-macosx/platform-tools/aidl] [-p/Users/neveu/Dev/android-sdk-macosx/platforms/android-15/framework.aidl] [-I/Users/neveu/Dev/magic_android/magic/src] [-I/Users/neveu/Dev/magic_android/src] [-I/Users/neveu/Dev/magic_android/3rdParty/Tapjoy] [-I/Users/neveu/Dev/magic_android/gen] [/Users/neveu/Dev/magic_android/src/com/android/vending/billing/IMarketBillingService.aidl] [/Users/neveu/Dev/magic_android/gen/com/android/vending/billing/IMarketBillingService.java] 2012-11-04 20:41:14,004 [ 24200] INFO - dom.attrs.AttributeDefinitions - Found tag with unknown parent: AndroidManifest.AndroidManifestCompatibleScreens 2012-11-04 20:41:18,781 [ 28977] INFO - s.impl.stores.FileBasedStorage - Document was not loaded for $APP_CONFIG$/cachedDictionary.xml file is null 2012-11-04 20:41:18,782 [ 28978] INFO - .impl.stores.XmlElementStorage - Document was not loaded for $APP_CONFIG$/cachedDictionary.xml

  Read the article

• Mysql - help me optimize this query

  - by sandeepan-nath

  About the system: -The system has a total of 8 tables - Users - Tutor_Details (Tutors are a type of User,Tutor_Details table is linked to Users) - learning_packs, (stores packs created by tutors) - learning_packs_tag_relations, (holds tag relations meant for search) - tutors_tag_relations and tags and orders (containing purchase details of tutor's packs), order_details linked to orders and tutor_details. For a more clear idea about the tables involved please check the The tables section in the end. -A tags based search approach is being followed.Tag relations are created when new tutors register and when tutors create packs (this makes tutors and packs searcheable). For details please check the section How tags work in this system? below. Following is a simpler representation (not the actual) of the more complex query which I am trying to optimize:- I have used statements like explanation of parts in the query select SUM(DISTINCT( t.tag LIKE "%Dictatorship%" )) as key_1_total_matches, SUM(DISTINCT( t.tag LIKE "%democracy%" )) as key_2_total_matches, td., u., count(distinct(od.id_od)), if (lp.id_lp > 0) then some conditional logic on lp fields else 0 as tutor_popularity from Tutor_Details AS td JOIN Users as u on u.id_user = td.id_user LEFT JOIN Learning_Packs_Tag_Relations AS lptagrels ON td.id_tutor = lptagrels.id_tutor LEFT JOIN Learning_Packs AS lp ON lptagrels.id_lp = lp.id_lp LEFT JOIN `some other tables on lp.id_lp - let's call learning pack tables set (including Learning_Packs table)` LEFT JOIN Order_Details as od on td.id_tutor = od.id_author LEFT JOIN Orders as o on od.id_order = o.id_order LEFT JOIN Tutors_Tag_Relations as ttagrels ON td.id_tutor = ttagrels.id_tutor JOIN Tags as t on (t.id_tag = ttagrels.id_tag) OR (t.id_tag = lptagrels.id_tag) where some condition on Users table's fields AND CASE WHEN ((t.id_tag = lptagrels.id_tag) AND (lp.id_lp 0)) THEN `some conditions on learning pack tables set` ELSE 1 END AND CASE WHEN ((t.id_tag = wtagrels.id_tag) AND (wc.id_wc 0)) THEN `some conditions on webclasses tables set` ELSE 1 END AND CASE WHEN (od.id_od0) THEN od.id_author = td.id_tutor and some conditions on Orders table's fields ELSE 1 END AND ( t.tag LIKE "%Dictatorship%" OR t.tag LIKE "%democracy%") group by td.id_tutor HAVING key_1_total_matches = 1 AND key_2_total_matches = 1 order by tutor_popularity desc, u.surname asc, u.name asc limit 0,20 ===================================================================== What does the above query do? Does AND logic search on the search keywords (2 in this example - "Democracy" and "Dictatorship"). Returns only those tutors for which both the keywords are present in the union of the two sets - tutors details and details of all the packs created by a tutor. To make things clear - Suppose a Tutor name "Sandeepan Nath" has created a pack "My first pack", then:- Searching "Sandeepan Nath" returns Sandeepan Nath. Searching "Sandeepan first" returns Sandeepan Nath. Searching "Sandeepan second" does not return Sandeepan Nath. ====================================================================================== The problem The results returned by the above query are correct (AND logic working as per expectation), but the time taken by the query on heavily loaded databases is like 25 seconds as against normal query timings of the order of 0.005 - 0.0002 seconds, which makes it totally unusable. It is possible that some of the delay is being caused because all the possible fields have not yet been indexed, but I would appreciate a better query as a solution, optimized as much as possible, displaying the same results ========================================================================================== How tags work in this system? When a tutor registers, tags are entered and tag relations are created with respect to tutor's details like name, surname etc. When a Tutors create packs, again tags are entered and tag relations are created with respect to pack's details like pack name, description etc. tag relations for tutors stored in tutors_tag_relations and those for packs stored in learning_packs_tag_relations. All individual tags are stored in tags table. ==================================================================== The tables Most of the following tables contain many other fields which I have omitted here. CREATE TABLE IF NOT EXISTS users ( id_user int(10) unsigned NOT NULL AUTO_INCREMENT, name varchar(100) NOT NULL DEFAULT '', surname varchar(155) NOT NULL DEFAULT '', PRIMARY KEY (id_user) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=636 ; CREATE TABLE IF NOT EXISTS tutor_details ( id_tutor int(10) NOT NULL AUTO_INCREMENT, id_user int(10) NOT NULL DEFAULT '0', PRIMARY KEY (id_tutor), KEY Users_FKIndex1 (id_user) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=51 ; CREATE TABLE IF NOT EXISTS orders ( id_order int(10) unsigned NOT NULL AUTO_INCREMENT, PRIMARY KEY (id_order), KEY Orders_FKIndex1 (id_user), ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=275 ; ALTER TABLE orders ADD CONSTRAINT Orders_ibfk_1 FOREIGN KEY (id_user) REFERENCES users (id_user) ON DELETE NO ACTION ON UPDATE NO ACTION; CREATE TABLE IF NOT EXISTS order_details ( id_od int(10) unsigned NOT NULL AUTO_INCREMENT, id_order int(10) unsigned NOT NULL DEFAULT '0', id_author int(10) NOT NULL DEFAULT '0', PRIMARY KEY (id_od), KEY Order_Details_FKIndex1 (id_order) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=284 ; ALTER TABLE order_details ADD CONSTRAINT Order_Details_ibfk_1 FOREIGN KEY (id_order) REFERENCES orders (id_order) ON DELETE NO ACTION ON UPDATE NO ACTION; CREATE TABLE IF NOT EXISTS learning_packs ( id_lp int(10) unsigned NOT NULL AUTO_INCREMENT, id_author int(10) unsigned NOT NULL DEFAULT '0', PRIMARY KEY (id_lp), KEY Learning_Packs_FKIndex2 (id_author), KEY id_lp (id_lp) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=23 ; CREATE TABLE IF NOT EXISTS tags ( id_tag int(10) unsigned NOT NULL AUTO_INCREMENT, tag varchar(255) DEFAULT NULL, PRIMARY KEY (id_tag), UNIQUE KEY tag (tag), KEY id_tag (id_tag), KEY tag_2 (tag), KEY tag_3 (tag) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3419 ; CREATE TABLE IF NOT EXISTS tutors_tag_relations ( id_tag int(10) unsigned NOT NULL DEFAULT '0', id_tutor int(10) DEFAULT NULL, KEY Tutors_Tag_Relations (id_tag), KEY id_tutor (id_tutor), KEY id_tag (id_tag) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; ALTER TABLE tutors_tag_relations ADD CONSTRAINT Tutors_Tag_Relations_ibfk_1 FOREIGN KEY (id_tag) REFERENCES tags (id_tag) ON DELETE NO ACTION ON UPDATE NO ACTION; CREATE TABLE IF NOT EXISTS learning_packs_tag_relations ( id_tag int(10) unsigned NOT NULL DEFAULT '0', id_tutor int(10) DEFAULT NULL, id_lp int(10) unsigned DEFAULT NULL, KEY Learning_Packs_Tag_Relations_FKIndex1 (id_tag), KEY id_lp (id_lp), KEY id_tag (id_tag) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; ALTER TABLE learning_packs_tag_relations ADD CONSTRAINT Learning_Packs_Tag_Relations_ibfk_1 FOREIGN KEY (id_tag) REFERENCES tags (id_tag) ON DELETE NO ACTION ON UPDATE NO ACTION; =================================================================================== Following is the exact query (this includes classes also - tutors can create classes and search terms are matched with classes created by tutors):- select count(distinct(od.id_od)) as tutor_popularity, CASE WHEN (IF((wc.id_wc 0), ( wc.wc_api_status = 1 AND wc.wc_type = 0 AND wc.class_date '2010-06-01 22:00:56' AND wccp.status = 1 AND (wccp.country_code='IE' or wccp.country_code IN ('INT'))), 0)) THEN 1 ELSE 0 END as 'classes_published', CASE WHEN (IF((lp.id_lp 0), (lp.id_status = 1 AND lp.published = 1 AND lpcp.status = 1 AND (lpcp.country_code='IE' or lpcp.country_code IN ('INT'))),0)) THEN 1 ELSE 0 END as 'packs_published', td . * , u . * from Tutor_Details AS td JOIN Users as u on u.id_user = td.id_user LEFT JOIN Learning_Packs_Tag_Relations AS lptagrels ON td.id_tutor = lptagrels.id_tutor LEFT JOIN Learning_Packs AS lp ON lptagrels.id_lp = lp.id_lp LEFT JOIN Learning_Packs_Categories AS lpc ON lpc.id_lp_cat = lp.id_lp_cat LEFT JOIN Learning_Packs_Categories AS lpcp ON lpcp.id_lp_cat = lpc.id_parent LEFT JOIN Learning_Pack_Content as lpct on (lp.id_lp = lpct.id_lp) LEFT JOIN Webclasses_Tag_Relations AS wtagrels ON td.id_tutor = wtagrels.id_tutor LEFT JOIN WebClasses AS wc ON wtagrels.id_wc = wc.id_wc LEFT JOIN Learning_Packs_Categories AS wcc ON wcc.id_lp_cat = wc.id_wp_cat LEFT JOIN Learning_Packs_Categories AS wccp ON wccp.id_lp_cat = wcc.id_parent LEFT JOIN Order_Details as od on td.id_tutor = od.id_author LEFT JOIN Orders as o on od.id_order = o.id_order LEFT JOIN Tutors_Tag_Relations as ttagrels ON td.id_tutor = ttagrels.id_tutor JOIN Tags as t on (t.id_tag = ttagrels.id_tag) OR (t.id_tag = lptagrels.id_tag) OR (t.id_tag = wtagrels.id_tag) where (u.country='IE' or u.country IN ('INT')) AND CASE WHEN ((t.id_tag = lptagrels.id_tag) AND (lp.id_lp 0)) THEN lp.id_status = 1 AND lp.published = 1 AND lpcp.status = 1 AND (lpcp.country_code='IE' or lpcp.country_code IN ('INT')) ELSE 1 END AND CASE WHEN ((t.id_tag = wtagrels.id_tag) AND (wc.id_wc 0)) THEN wc.wc_api_status = 1 AND wc.wc_type = 0 AND wc.class_date '2010-06-01 22:00:56' AND wccp.status = 1 AND (wccp.country_code='IE' or wccp.country_code IN ('INT')) ELSE 1 END AND CASE WHEN (od.id_od0) THEN od.id_author = td.id_tutor and o.order_status = 'paid' and CASE WHEN (od.id_wc 0) THEN od.can_attend_class=1 ELSE 1 END ELSE 1 END AND 1 group by td.id_tutor order by tutor_popularity desc, u.surname asc, u.name asc limit 0,20 Please note - The provided database structure does not show all the fields and tables as in this query

  Read the article

• How do I control the direction of the scroll on my coda slider?

  - by lightingwrist

  Hello, I have a coda slider and am unable to determine the direction of the scroll. I have 3 panels that I want to scroll left to right. Sometimes it scrolls left to right, sometimes up and down, and sometimes horizontally. How do I lock it down to go the direction I want? Here is the HTML: <body> <div id="slider_home" class="round_10px"> <ul class="navigation_home"> <li><a href="#scroll_Parents" class="round_10px">Information For Parents</a></li> <li><a href="#scroll_Materials" class="round_10px">Print Materials</a></li> <li><a href="#scroll_Resources" class="round_10px">Online Resources</a></li> </ul> <div id="scroll_bg_home"> <div class="scroll_home"> <div class="scrollContainer_home"> <div class="panel_home" id="scroll_Parents"> content </div> <div class="panel_home" id="scroll_Materials"> content </div> <div class="panel_home" id="scroll_Resources"> content </div> </div> </div> </div> </div> </body> Here is the CSS: #wrapper {width:550px;margin:0px auto;} #intro {padding-bottom:10px;} h2 {margin:0;margin-bottom:14px;padding:0;} #slider {width:631px;margin:10px auto 0px auto;position:relative;} #scroll_bg{height:360px;width:590px;overflow:hidden;position:relative;clear:left;background:#FFFFFF url(images/) no-repeat; margin:0px auto 0px auto} .scroll{ background:transparent; width:550px; height:370px; padding:0px; margin:0px auto; overflow:hidden; } .scrollContainer div.panel {padding:20px 0px;height:330px; width:550px;margin:0px;float:left;} #shade {background:#EDEDEC url(images/shade.jpg) no-repeat 0 0;height:50px;} ul.navigation {list-style:none;margin:0px 0px 0px 23px;padding:0px;padding-bottom:0px;} ul.navigation li {display:inline; margin-right:5px;} ul.navigation li a { background:#FFFFFF;font-family:Arial, Helvetica, sans-serif; font-size:16px; font-weight:bold; color:#CCCCCC;padding:5px 5px 5px 5px;border:1px #F4F4F4 solid;text-decoration: none;} ul.navigation a:hover { color:#EDEDEC;border:1px #E6E6E6 solid;} ul.navigation a.selected {color:#333333;} ul.navigation a:focus {outline:none;} .scrollButtons {position:absolute;top:150px;cursor:pointer;} .scrollButtons.left {left:-37px;top:20px} .scrollButtons.right {right:-32px;top:20px;} .hide {display:none;} And here is the Jquery includes file: // when the DOM is ready... $(document).ready(function () { var $panels = $('#slider_home .scrollContainer_home > div.panel_home'); var $container = $('#slider_home .scrollContainer_home'); // if true, we'll float all the panels left and fix the width // of the container var horizontal = true; // float the panels left if we're going horizontal if (horizontal) { $panels.css({ 'float' : 'left', 'position' : 'relative' // IE fix to ensure overflow is hidden }); // calculate a new width for the container (so it holds all panels) $container.css('width', $panels[0].offsetWidth * $panels.length); } // collect the scroll object, at the same time apply the hidden overflow // to remove the default scrollbars that will appear var $scroll_bg = $('#scroll_bg_home'); var $scroll = $('#slider_home .scroll_home').css('overflow', 'hidden'); // apply our left + right buttons $scroll_bg .before('<img class="scrollButtons_home left" src="styles/images/BackFlip.jpg" />') .after('<img class="scrollButtons_home right" src="styles/images/flipForward.jpg" />'); // handle nav selection function selectNav() { $(this) .parents('ul:first') .find('a') .removeClass('selected') .end() .end() .addClass('selected'); } $('.navigation_home').find('a').click(selectNav); // go find the navigation link that has this target and select the nav function trigger(data) { var el = $('.navigation_home').find('a[href$="' + data.id + '"]').get(0); selectNav.call(el); } if (window.location.hash) { trigger({ id : window.location.hash.substr(1) }); } else { $('.navigation_home a:first').click(); } // offset is used to move to *exactly* the right place, since I'm using // padding on my example, I need to subtract the amount of padding to // the offset. Try removing this to get a good idea of the effect var offset = parseInt((horizontal ? $container.css('paddingTop') : $container.css('paddingLeft')) || 0) * -1; var scrollOptions = { target: $scroll, // the element that has the overflow // can be a selector which will be relative to the target items: $panels, navigation: '.navigation_home a', // selectors are NOT relative to document, i.e. make sure they're unique prev: 'img.left', next: 'img.right', // allow the scroll effect to run both directions axis: 'xy', onAfter: trigger, // our final callback offset: offset, // duration of the sliding effect duration: 500, // easing - can be used with the easing plugin: // http://gsgd.co.uk/sandbox/jquery/easing/ easing: 'swing' }; // apply serialScroll to the slider - we chose this plugin because it // supports// the indexed next and previous scroll along with hooking // in to our navigation. $('#slider_home').serialScroll(scrollOptions); // now apply localScroll to hook any other arbitrary links to trigger // the effect $.localScroll(scrollOptions); // finally, if the URL has a hash, move the slider in to position, // setting the duration to 1 because I don't want it to scroll in the // very first page load. We don't always need this, but it ensures // the positioning is absolutely spot on when the pages loads. scrollOptions.duration = 1; $.localScroll.hash(scrollOptions); });

  Read the article

• User controls Stopped working after Migration from 3.7 to 5.2

  - by user1400290

  I recently Migrated my 3.7 sp4 project to 5.2, but I had issues while doing so. Currently, my user controls are not working after migration in 5.2 project. Below is the code: User Control Code: <%@ Control Language="C#" AutoEventWireup="true" CodeFile="SiteMenu.ascx.cs" Inherits="UserControls_Nav_SiteMenu" %> <%@ Register TagPrefix="telerik" Assembly="Telerik.Web.UI" Namespace="Telerik.Web.UI" %> <asp:SiteMapDataSource ID="SiteMapDataSource1" runat="server" ShowStartingNode="false" /> <telerik:RadMenu ID="RadMenu1" runat="server" DataSourceID="SitemapDataSource1" OnItemDataBound="RadMenu1_ItemDataBound"> </telerik:RadMenu> User Control's Class code: using System; using System.Data; using System.Configuration; using System.Collections; using System.Web; using System.Web.Security; using System.Web.UI; using System.Web.UI.WebControls; using System.Web.UI.WebControls.WebParts; using System.Web.UI.HtmlControls; using System.ComponentModel; using System.Drawing; using Telerik; using Telerik.Cms; using Telerik.Cms.Web; using Telerik.Web.UI; using Telerik.Caching; using Telerik.Cms.Web.UI; [DefaultProperty("StartingNodeOffset")] public partial class UserControls_Nav_SiteMenu : System.Web.UI.UserControl, ICacheableObject { protected void Page_Load(object sender, EventArgs e) { } protected override void Render(HtmlTextWriter writer) { // Checks if this is called by the Search Indexer and does not render anything if so. // Navigation controls are present in every page and should NOT be indexed multiple times. if (!CmsContext.IsRequestCrawler(this.Context)) base.Render(writer); } #region Data Fields private bool hideUrlForGroupPages = false; private string selectedItemCssClass = "selectedItem"; #endregion #region Properties [Browsable(true)] [Category("Behavior")] public int LastExpandLevel { get { if (this.RadMenu1.MaxDataBindDepth < 0) return 0; return this.RadMenu1.MaxDataBindDepth; } set { if (value == 0) this.RadMenu1.MaxDataBindDepth = -1; else this.RadMenu1.MaxDataBindDepth = value; } } [Browsable(true)] [Category("Behavior")] public int ExpandDelay { get { return this.RadMenu1.ExpandDelay; } set { this.RadMenu1.ExpandDelay = value; } } [Browsable(true)] [Category("Behavior")] public bool ClickToOpen { get { return this.RadMenu1.ClickToOpen; } set { this.RadMenu1.ClickToOpen = value; } } [Browsable(true)] [Category("Behavior")] [DefaultValue(false)] public bool HideUrlForGroupPages { get { return this.hideUrlForGroupPages; } set { this.hideUrlForGroupPages = value; } } [Browsable(true)] [Category("Appearance")] public string SelectedItemCssClass { get { return this.selectedItemCssClass; } set { this.selectedItemCssClass = value; } } [Browsable(true)] [Category("Appearance")] public string CssClass { get { return this.RadMenu1.CssClass; } set { this.RadMenu1.CssClass = value; } } [Browsable(true)] public RadMenu Menu { get { return this.RadMenu1; } set { this.RadMenu1 = value; } } [Browsable(true)] [Category("Navigation")] public int StartingNodeOffset { get { return this.SiteMapDataSource1.StartingNodeOffset; } set { this.SiteMapDataSource1.StartingNodeOffset = value; } } [WebEditor("Telerik.Cms.Web.UI.UrlEditorWrapper, Telerik.Cms")] [Browsable(true)] [Category("Navigation")] public string StartingNodeUrl { get { return this.SiteMapDataSource1.StartingNodeUrl; } set { this.SiteMapDataSource1.StartingNodeUrl = value; } } [Browsable(true)] [Category("Navigation")] public bool StartFromCurrentNode { get { return this.SiteMapDataSource1.StartFromCurrentNode; } set { this.SiteMapDataSource1.StartFromCurrentNode = value; } } [Browsable(true)] [Category("Navigation")] public bool ShowStartingNode { get { return this.SiteMapDataSource1.ShowStartingNode; } set { this.SiteMapDataSource1.ShowStartingNode = value; } } /// <summary>(Exposed from contained RadMenu.)</summary> [Browsable(true)] [Category("Appearance")] public string SkinID { get { return this.RadMenu1.SkinID; } set { this.RadMenu1.SkinID = value; } } [Browsable(true)] [Category("Appearance")] public string Skin { get { return this.RadMenu1.Skin; } set { this.RadMenu1.Skin = value; } } #endregion #region Methods public void RadMenu1_ItemDataBound(object sender, RadMenuEventArgs e) { CmsSiteMapNode node = e.Item.DataItem as CmsSiteMapNode; if (this.hideUrlForGroupPages) { if (node != null) { // save the PageID in the attributes of the menu item e.Item.Attributes.Add("PageID", node.Key); if (node.PageType == CmsPageType.Group) { e.Item.NavigateUrl = ""; } } } if (node.CmsPage != null) { if (node.CmsPage.PageType == CmsPageType.External) { e.Item.Target = "_blank"; } } } #endregion #region ICacheableObject Members public System.Web.Caching.CacheDependency[] GetDependencies() { CmsSiteMapProvider provider = null; if (!String.IsNullOrEmpty(this.SiteMapDataSource1.SiteMapProvider)) provider = SiteMap.Providers[this.SiteMapDataSource1.SiteMapProvider] as CmsSiteMapProvider; else provider = SiteMap.Provider as CmsSiteMapProvider; if (provider != null) { return new System.Web.Caching.CacheDependency[]{ provider.CloneCacheDependency()}; } return null; } #endregion } When I edit the Template(in Admin mode), the following error is displayed in control location: Both DataSource and DataSourceID are defined on 'RadMenu1'. Remove one definition. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.InvalidOperationException: Both DataSource and DataSourceID are defined on 'RadMenu1'. Remove one definition. Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. Stack Trace: [InvalidOperationException: Both DataSource and DataSourceID are defined on 'RadMenu1'. Remove one definition.] System.Web.UI.WebControls.DataBoundControl.ConnectToDataSourceView() +3234866 System.Web.UI.WebControls.DataBoundControl.OnLoad(EventArgs e) +28 System.Web.UI.Control.LoadRecursive() +71 System.Web.UI.Control.LoadRecursive() +190 System.Web.UI.Control.LoadRecursive() +190 System.Web.UI.Control.AddedControl(Control control, Int32 index) +11422584 System.Web.UI.Control.EnsureChildControls() +182 System.Web.UI.Control.PreRenderRecursiveInternal() +60 System.Web.UI.Control.PreRenderRecursiveInternal() +222 System.Web.UI.Control.PreRenderRecursiveInternal() +222 System.Web.UI.Control.PreRenderRecursiveInternal() +222 System.Web.UI.Control.PreRenderRecursiveInternal() +222 System.Web.UI.Control.PreRenderRecursiveInternal() +222 System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4201 Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.272 but I searched in my code as you can see above there's only DataSourceID is defined. What should I do? Thanks

  Read the article

• Using R to Analyze G1GC Log Files

  - by user12620111

  Using R to Analyze G1GC Log Files body, td { font-family: sans-serif; background-color: white; font-size: 12px; margin: 8px; } tt, code, pre { font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; } h1 { font-size:2.2em; } h2 { font-size:1.8em; } h3 { font-size:1.4em; } h4 { font-size:1.0em; } h5 { font-size:0.9em; } h6 { font-size:0.8em; } a:visited { color: rgb(50%, 0%, 50%); } pre { margin-top: 0; max-width: 95%; border: 1px solid #ccc; white-space: pre-wrap; } pre code { display: block; padding: 0.5em; } code.r, code.cpp { background-color: #F8F8F8; } table, td, th { border: none; } blockquote { color:#666666; margin:0; padding-left: 1em; border-left: 0.5em #EEE solid; } hr { height: 0px; border-bottom: none; border-top-width: thin; border-top-style: dotted; border-top-color: #999999; } @media print { * { background: transparent !important; color: black !important; filter:none !important; -ms-filter: none !important; } body { font-size:12pt; max-width:100%; } a, a:visited { text-decoration: underline; } hr { visibility: hidden; page-break-before: always; } pre, blockquote { padding-right: 1em; page-break-inside: avoid; } tr, img { page-break-inside: avoid; } img { max-width: 100% !important; } @page :left { margin: 15mm 20mm 15mm 10mm; } @page :right { margin: 15mm 10mm 15mm 20mm; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3 { page-break-after: avoid; } } pre .operator, pre .paren { color: rgb(104, 118, 135) } pre .literal { color: rgb(88, 72, 246) } pre .number { color: rgb(0, 0, 205); } pre .comment { color: rgb(76, 136, 107); } pre .keyword { color: rgb(0, 0, 255); } pre .identifier { color: rgb(0, 0, 0); } pre .string { color: rgb(3, 106, 7); } var hljs=new function(){function m(p){return p.replace(/&/gm,"&").replace(/"}while(y.length||w.length){var v=u().splice(0,1)[0];z+=m(x.substr(q,v.offset-q));q=v.offset;if(v.event=="start"){z+=t(v.node);s.push(v.node)}else{if(v.event=="stop"){var p,r=s.length;do{r--;p=s[r];z+=("")}while(p!=v.node);s.splice(r,1);while(r'+M[0]+""}else{r+=M[0]}O=P.lR.lastIndex;M=P.lR.exec(L)}return r+L.substr(O,L.length-O)}function J(L,M){if(M.sL&&e[M.sL]){var r=d(M.sL,L);x+=r.keyword_count;return r.value}else{return F(L,M)}}function I(M,r){var L=M.cN?'':"";if(M.rB){y+=L;M.buffer=""}else{if(M.eB){y+=m(r)+L;M.buffer=""}else{y+=L;M.buffer=r}}D.push(M);A+=M.r}function G(N,M,Q){var R=D[D.length-1];if(Q){y+=J(R.buffer+N,R);return false}var P=q(M,R);if(P){y+=J(R.buffer+N,R);I(P,M);return P.rB}var L=v(D.length-1,M);if(L){var O=R.cN?"":"";if(R.rE){y+=J(R.buffer+N,R)+O}else{if(R.eE){y+=J(R.buffer+N,R)+O+m(M)}else{y+=J(R.buffer+N+M,R)+O}}while(L1){O=D[D.length-2].cN?"":"";y+=O;L--;D.length--}var r=D[D.length-1];D.length--;D[D.length-1].buffer="";if(r.starts){I(r.starts,"")}return R.rE}if(w(M,R)){throw"Illegal"}}var E=e[B];var D=[E.dM];var A=0;var x=0;var y="";try{var s,u=0;E.dM.buffer="";do{s=p(C,u);var t=G(s[0],s[1],s[2]);u+=s[0].length;if(!t){u+=s[1].length}}while(!s[2]);if(D.length1){throw"Illegal"}return{r:A,keyword_count:x,value:y}}catch(H){if(H=="Illegal"){return{r:0,keyword_count:0,value:m(C)}}else{throw H}}}function g(t){var p={keyword_count:0,r:0,value:m(t)};var r=p;for(var q in e){if(!e.hasOwnProperty(q)){continue}var s=d(q,t);s.language=q;if(s.keyword_count+s.rr.keyword_count+r.r){r=s}if(s.keyword_count+s.rp.keyword_count+p.r){r=p;p=s}}if(r.language){p.second_best=r}return p}function i(r,q,p){if(q){r=r.replace(/^((]+|\t)+)/gm,function(t,w,v,u){return w.replace(/\t/g,q)})}if(p){r=r.replace(/\n/g,"")}return r}function n(t,w,r){var x=h(t,r);var v=a(t);var y,s;if(v){y=d(v,x)}else{return}var q=c(t);if(q.length){s=document.createElement("pre");s.innerHTML=y.value;y.value=k(q,c(s),x)}y.value=i(y.value,w,r);var u=t.className;if(!u.match("(\\s|^)(language-)?"+v+"(\\s|$)")){u=u?(u+" "+v):v}if(/MSIE [678]/.test(navigator.userAgent)&&t.tagName=="CODE"&&t.parentNode.tagName=="PRE"){s=t.parentNode;var p=document.createElement("div");p.innerHTML=""+y.value+"";t=p.firstChild.firstChild;p.firstChild.cN=s.cN;s.parentNode.replaceChild(p.firstChild,s)}else{t.innerHTML=y.value}t.className=u;t.result={language:v,kw:y.keyword_count,re:y.r};if(y.second_best){t.second_best={language:y.second_best.language,kw:y.second_best.keyword_count,re:y.second_best.r}}}function o(){if(o.called){return}o.called=true;var r=document.getElementsByTagName("pre");for(var p=0;p|=||=||=|\\?|\\[|\\{|\\(|\\^|\\^=|\\||\\|=|\\|\\||~";this.ER="(?![\\s\\S])";this.BE={b:"\\\\.",r:0};this.ASM={cN:"string",b:"'",e:"'",i:"\\n",c:[this.BE],r:0};this.QSM={cN:"string",b:'"',e:'"',i:"\\n",c:[this.BE],r:0};this.CLCM={cN:"comment",b:"//",e:"$"};this.CBLCLM={cN:"comment",b:"/\\*",e:"\\*/"};this.HCM={cN:"comment",b:"#",e:"$"};this.NM={cN:"number",b:this.NR,r:0};this.CNM={cN:"number",b:this.CNR,r:0};this.BNM={cN:"number",b:this.BNR,r:0};this.inherit=function(r,s){var p={};for(var q in r){p[q]=r[q]}if(s){for(var q in s){p[q]=s[q]}}return p}}();hljs.LANGUAGES.cpp=function(){var a={keyword:{"false":1,"int":1,"float":1,"while":1,"private":1,"char":1,"catch":1,"export":1,virtual:1,operator:2,sizeof:2,dynamic_cast:2,typedef:2,const_cast:2,"const":1,struct:1,"for":1,static_cast:2,union:1,namespace:1,unsigned:1,"long":1,"throw":1,"volatile":2,"static":1,"protected":1,bool:1,template:1,mutable:1,"if":1,"public":1,friend:2,"do":1,"return":1,"goto":1,auto:1,"void":2,"enum":1,"else":1,"break":1,"new":1,extern:1,using:1,"true":1,"class":1,asm:1,"case":1,typeid:1,"short":1,reinterpret_cast:2,"default":1,"double":1,register:1,explicit:1,signed:1,typename:1,"try":1,"this":1,"switch":1,"continue":1,wchar_t:1,inline:1,"delete":1,alignof:1,char16_t:1,char32_t:1,constexpr:1,decltype:1,noexcept:1,nullptr:1,static_assert:1,thread_local:1,restrict:1,_Bool:1,complex:1},built_in:{std:1,string:1,cin:1,cout:1,cerr:1,clog:1,stringstream:1,istringstream:1,ostringstream:1,auto_ptr:1,deque:1,list:1,queue:1,stack:1,vector:1,map:1,set:1,bitset:1,multiset:1,multimap:1,unordered_set:1,unordered_map:1,unordered_multiset:1,unordered_multimap:1,array:1,shared_ptr:1}};return{dM:{k:a,i:"",k:a,r:10,c:["self"]}]}}}();hljs.LANGUAGES.r={dM:{c:[hljs.HCM,{cN:"number",b:"\\b0[xX][0-9a-fA-F]+[Li]?\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\b\\d+(?:[eE][+\\-]?\\d*)?L\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\b\\d+\\.(?!\\d)(?:i\\b)?",e:hljs.IMMEDIATE_RE,r:1},{cN:"number",b:"\\b\\d+(?:\\.\\d*)?(?:[eE][+\\-]?\\d*)?i?\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\.\\d+(?:[eE][+\\-]?\\d*)?i?\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"keyword",b:"(?:tryCatch|library|setGeneric|setGroupGeneric)\\b",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\.\\.\\.",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\.\\.\\d+(?![\\w.])",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\b(?:function)",e:hljs.IMMEDIATE_RE,r:2},{cN:"keyword",b:"(?:if|in|break|next|repeat|else|for|return|switch|while|try|stop|warning|require|attach|detach|source|setMethod|setClass)\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"literal",b:"(?:NA|NA_integer_|NA_real_|NA_character_|NA_complex_)\\b",e:hljs.IMMEDIATE_RE,r:10},{cN:"literal",b:"(?:NULL|TRUE|FALSE|T|F|Inf|NaN)\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"identifier",b:"[a-zA-Z.][a-zA-Z0-9._]*\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"operator",b:"|=||   Using R to Analyze G1GC Log Files   Using R to Analyze G1GC Log Files Introduction Working in Oracle Platform Integration gives an engineer opportunities to work on a wide array of technologies. My team’s goal is to make Oracle applications run best on the Solaris/SPARC platform. When looking for bottlenecks in a modern applications, one needs to be aware of not only how the CPUs and operating system are executing, but also network, storage, and in some cases, the Java Virtual Machine. I was recently presented with about 1.5 GB of Java Garbage First Garbage Collector log file data. If you’re not familiar with the subject, you might want to review Garbage First Garbage Collector Tuning by Monica Beckwith. The customer had been running Java HotSpot 1.6.0_31 to host a web application server. I was told that the Solaris/SPARC server was running a Java process launched using a commmand line that included the following flags: -d64 -Xms9g -Xmx9g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=80 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintFlagsFinal -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:ParallelGCThreads=8 Several sources on the internet indicate that if I were to print out the 1.5 GB of log files, it would require enough paper to fill the bed of a pick up truck. Of course, it would be fruitless to try to scan the log files by hand. Tools will be required to summarize the contents of the log files. Others have encountered large Java garbage collection log files. There are existing tools to analyze the log files: IBM’s GC toolkit The chewiebug GCViewer gchisto HPjmeter Instead of using one of the other tools listed, I decide to parse the log files with standard Unix tools, and analyze the data with R. Data Cleansing The log files arrived in two different formats. I guess that the difference is that one set of log files was generated using a more verbose option, maybe -XX:+PrintHeapAtGC, and the other set of log files was generated without that option. Format 1 In some of the log files, the log files with the less verbose format, a single trace, i.e. the report of a singe garbage collection event, looks like this: {Heap before GC invocations=12280 (full 61): garbage-first heap total 9437184K, used 7499918K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) region size 4096K, 1 young (4096K), 0 survivors (0K) compacting perm gen total 262144K, used 144077K [0xffffffff40000000, 0xffffffff50000000, 0xffffffff50000000) the space 262144K, 54% used [0xffffffff40000000, 0xffffffff48cb3758, 0xffffffff48cb3800, 0xffffffff50000000) No shared spaces configured. 2014-05-14T07:24:00.988-0700: 60586.353: [GC pause (young) 7324M->7320M(9216M), 0.1567265 secs] Heap after GC invocations=12281 (full 61): garbage-first heap total 9437184K, used 7496533K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) region size 4096K, 0 young (0K), 0 survivors (0K) compacting perm gen total 262144K, used 144077K [0xffffffff40000000, 0xffffffff50000000, 0xffffffff50000000) the space 262144K, 54% used [0xffffffff40000000, 0xffffffff48cb3758, 0xffffffff48cb3800, 0xffffffff50000000) No shared spaces configured. } A simple grep can be used to extract a summary: $ grep "\[ GC pause (young" g1gc.log 2014-05-13T13:24:35.091-0700: 3.109: [GC pause (young) 20M->5029K(9216M), 0.0146328 secs] 2014-05-13T13:24:35.440-0700: 3.459: [GC pause (young) 9125K->6077K(9216M), 0.0086723 secs] 2014-05-13T13:24:37.581-0700: 5.599: [GC pause (young) 25M->8470K(9216M), 0.0203820 secs] 2014-05-13T13:24:42.686-0700: 10.704: [GC pause (young) 44M->15M(9216M), 0.0288848 secs] 2014-05-13T13:24:48.941-0700: 16.958: [GC pause (young) 51M->20M(9216M), 0.0491244 secs] 2014-05-13T13:24:56.049-0700: 24.066: [GC pause (young) 92M->26M(9216M), 0.0525368 secs] 2014-05-13T13:25:34.368-0700: 62.383: [GC pause (young) 602M->68M(9216M), 0.1721173 secs] But that format wasn't easily read into R, so I needed to be a bit more tricky. I used the following Unix command to create a summary file that was easy for R to read. $ echo "SecondsSinceLaunch BeforeSize AfterSize TotalSize RealTime" $ grep "\[GC pause (young" g1gc.log | grep -v mark | sed -e 's/[A-SU-z\(\),]/ /g' -e 's/->/ /' -e 's/: / /g' | more SecondsSinceLaunch BeforeSize AfterSize TotalSize RealTime 2014-05-13T13:24:35.091-0700 3.109 20 5029 9216 0.0146328 2014-05-13T13:24:35.440-0700 3.459 9125 6077 9216 0.0086723 2014-05-13T13:24:37.581-0700 5.599 25 8470 9216 0.0203820 2014-05-13T13:24:42.686-0700 10.704 44 15 9216 0.0288848 2014-05-13T13:24:48.941-0700 16.958 51 20 9216 0.0491244 2014-05-13T13:24:56.049-0700 24.066 92 26 9216 0.0525368 2014-05-13T13:25:34.368-0700 62.383 602 68 9216 0.1721173 Format 2 In some of the log files, the log files with the more verbose format, a single trace, i.e. the report of a singe garbage collection event, was more complicated than Format 1. Here is a text file with an example of a single G1GC trace in the second format. As you can see, it is quite complicated. It is nice that there is so much information available, but the level of detail can be overwhelming. I wrote this awk script (download) to summarize each trace on a single line. #!/usr/bin/env awk -f BEGIN { printf("SecondsSinceLaunch IncrementalCount FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize\n") } ###################### # Save count data from lines that are at the start of each G1GC trace. # Each trace starts out like this: # {Heap before GC invocations=14 (full 0): # garbage-first heap total 9437184K, used 325496K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) ###################### /{Heap.*full/{ gsub ( "\\)" , "" ); nf=split($0,a,"="); split(a[2],b," "); getline; if ( match($0, "first") ) { G1GC=1; IncrementalCount=b[1]; FullCount=substr( b[3], 1, length(b[3])-1 ); } else { G1GC=0; } } ###################### # Pull out time stamps that are in lines with this format: # 2014-05-12T14:02:06.025-0700: 94.312: [GC pause (young), 0.08870154 secs] ###################### /GC pause/ { DateTime=$1; SecondsSinceLaunch=substr($2, 1, length($2)-1); } ###################### # Heap sizes are in lines that look like this: # [ 4842M->4838M(9216M)] ###################### /\[ .*]$/ { gsub ( "\\[" , "" ); gsub ( "\ \]" , "" ); gsub ( "->" , " " ); gsub ( "\\( " , " " ); gsub ( "\ \)" , " " ); split($0,a," "); if ( split(a[1],b,"M") > 1 ) {BeforeSize=b[1]*1024;} if ( split(a[1],b,"K") > 1 ) {BeforeSize=b[1];} if ( split(a[2],b,"M") > 1 ) {AfterSize=b[1]*1024;} if ( split(a[2],b,"K") > 1 ) {AfterSize=b[1];} if ( split(a[3],b,"M") > 1 ) {TotalSize=b[1]*1024;} if ( split(a[3],b,"K") > 1 ) {TotalSize=b[1];} } ###################### # Emit an output line when you find input that looks like this: # [Times: user=1.41 sys=0.08, real=0.24 secs] ###################### /\[Times/ { if (G1GC==1) { gsub ( "," , "" ); split($2,a,"="); UserTime=a[2]; split($3,a,"="); SysTime=a[2]; split($4,a,"="); RealTime=a[2]; print DateTime,SecondsSinceLaunch,IncrementalCount,FullCount,UserTime,SysTime,RealTime,BeforeSize,AfterSize,TotalSize; G1GC=0; } } The resulting summary is about 25X smaller that the original file, but still difficult for a human to digest. SecondsSinceLaunch IncrementalCount FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ... 2014-05-12T18:36:34.669-0700: 3985.744 561 0 0.57 0.06 0.16 1724416 1720320 9437184 2014-05-12T18:36:34.839-0700: 3985.914 562 0 0.51 0.06 0.19 1724416 1720320 9437184 2014-05-12T18:36:35.069-0700: 3986.144 563 0 0.60 0.04 0.27 1724416 1721344 9437184 2014-05-12T18:36:35.354-0700: 3986.429 564 0 0.33 0.04 0.09 1725440 1722368 9437184 2014-05-12T18:36:35.545-0700: 3986.620 565 0 0.58 0.04 0.17 1726464 1722368 9437184 2014-05-12T18:36:35.726-0700: 3986.801 566 0 0.43 0.05 0.12 1726464 1722368 9437184 2014-05-12T18:36:35.856-0700: 3986.930 567 0 0.30 0.04 0.07 1726464 1723392 9437184 2014-05-12T18:36:35.947-0700: 3987.023 568 0 0.61 0.04 0.26 1727488 1723392 9437184 2014-05-12T18:36:36.228-0700: 3987.302 569 0 0.46 0.04 0.16 1731584 1724416 9437184 Reading the Data into R Once the GC log data had been cleansed, either by processing the first format with the shell script, or by processing the second format with the awk script, it was easy to read the data into R. g1gc.df = read.csv("summary.txt", row.names = NULL, stringsAsFactors=FALSE,sep="") str(g1gc.df) ## 'data.frame': 8307 obs. of 10 variables: ## $ row.names : chr "2014-05-12T14:00:32.868-0700:" "2014-05-12T14:00:33.179-0700:" "2014-05-12T14:00:33.677-0700:" "2014-05-12T14:00:35.538-0700:" ... ## $ SecondsSinceLaunch: num 1.16 1.47 1.97 3.83 6.1 ... ## $ IncrementalCount : int 0 1 2 3 4 5 6 7 8 9 ... ## $ FullCount : int 0 0 0 0 0 0 0 0 0 0 ... ## $ UserTime : num 0.11 0.05 0.04 0.21 0.08 0.26 0.31 0.33 0.34 0.56 ... ## $ SysTime : num 0.04 0.01 0.01 0.05 0.01 0.06 0.07 0.06 0.07 0.09 ... ## $ RealTime : num 0.02 0.02 0.01 0.04 0.02 0.04 0.05 0.04 0.04 0.06 ... ## $ BeforeSize : int 8192 5496 5768 22528 24576 43008 34816 53248 55296 93184 ... ## $ AfterSize : int 1400 1672 2557 4907 7072 14336 16384 18432 19456 21504 ... ## $ TotalSize : int 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 ... head(g1gc.df) ## row.names SecondsSinceLaunch IncrementalCount ## 1 2014-05-12T14:00:32.868-0700: 1.161 0 ## 2 2014-05-12T14:00:33.179-0700: 1.472 1 ## 3 2014-05-12T14:00:33.677-0700: 1.969 2 ## 4 2014-05-12T14:00:35.538-0700: 3.830 3 ## 5 2014-05-12T14:00:37.811-0700: 6.103 4 ## 6 2014-05-12T14:00:41.428-0700: 9.720 5 ## FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ## 1 0 0.11 0.04 0.02 8192 1400 9437184 ## 2 0 0.05 0.01 0.02 5496 1672 9437184 ## 3 0 0.04 0.01 0.01 5768 2557 9437184 ## 4 0 0.21 0.05 0.04 22528 4907 9437184 ## 5 0 0.08 0.01 0.02 24576 7072 9437184 ## 6 0 0.26 0.06 0.04 43008 14336 9437184 Basic Statistics Once the data has been read into R, simple statistics are very easy to generate. All of the numbers from high school statistics are available via simple commands. For example, generate a summary of every column: summary(g1gc.df) ## row.names SecondsSinceLaunch IncrementalCount FullCount ## Length:8307 Min. : 1 Min. : 0 Min. : 0.0 ## Class :character 1st Qu.: 9977 1st Qu.:2048 1st Qu.: 0.0 ## Mode :character Median :12855 Median :4136 Median : 12.0 ## Mean :12527 Mean :4156 Mean : 31.6 ## 3rd Qu.:15758 3rd Qu.:6262 3rd Qu.: 61.0 ## Max. :55484 Max. :8391 Max. :113.0 ## UserTime SysTime RealTime BeforeSize ## Min. :0.040 Min. :0.0000 Min. : 0.0 Min. : 5476 ## 1st Qu.:0.470 1st Qu.:0.0300 1st Qu.: 0.1 1st Qu.:5137920 ## Median :0.620 Median :0.0300 Median : 0.1 Median :6574080 ## Mean :0.751 Mean :0.0355 Mean : 0.3 Mean :5841855 ## 3rd Qu.:0.920 3rd Qu.:0.0400 3rd Qu.: 0.2 3rd Qu.:7084032 ## Max. :3.370 Max. :1.5600 Max. :488.1 Max. :8696832 ## AfterSize TotalSize ## Min. : 1380 Min. :9437184 ## 1st Qu.:5002752 1st Qu.:9437184 ## Median :6559744 Median :9437184 ## Mean :5785454 Mean :9437184 ## 3rd Qu.:7054336 3rd Qu.:9437184 ## Max. :8482816 Max. :9437184 Q: What is the total amount of User CPU time spent in garbage collection? sum(g1gc.df$UserTime) ## [1] 6236 As you can see, less than two hours of CPU time was spent in garbage collection. Is that too much? To find the percentage of time spent in garbage collection, divide the number above by total_elapsed_time*CPU_count. In this case, there are a lot of CPU’s and it turns out the the overall amount of CPU time spent in garbage collection isn’t a problem when viewed in isolation. When calculating rates, i.e. events per unit time, you need to ask yourself if the rate is homogenous across the time period in the log file. Does the log file include spikes of high activity that should be separately analyzed? Averaging in data from nights and weekends with data from business hours may alias problems. If you have a reason to suspect that the garbage collection rates include peaks and valleys that need independent analysis, see the “Time Series” section, below. Q: How much garbage is collected on each pass? The amount of heap space that is recovered per GC pass is surprisingly low: At least one collection didn’t recover any data. (“Min.=0”) 25% of the passes recovered 3MB or less. (“1st Qu.=3072”) Half of the GC passes recovered 4MB or less. (“Median=4096”) The average amount recovered was 56MB. (“Mean=56390”) 75% of the passes recovered 36MB or less. (“3rd Qu.=36860”) At least one pass recovered 2GB. (“Max.=2121000”) g1gc.df$Delta = g1gc.df$BeforeSize - g1gc.df$AfterSize summary(g1gc.df$Delta) ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0 3070 4100 56400 36900 2120000 Q: What is the maximum User CPU time for a single collection? The worst garbage collection (“Max.”) is many standard deviations away from the mean. The data appears to be right skewed. summary(g1gc.df$UserTime) ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0.040 0.470 0.620 0.751 0.920 3.370 sd(g1gc.df$UserTime) ## [1] 0.3966 Basic Graphics Once the data is in R, it is trivial to plot the data with formats including dot plots, line charts, bar charts (simple, stacked, grouped), pie charts, boxplots, scatter plots histograms, and kernel density plots. Histogram of User CPU Time per Collection I don't think that this graph requires any explanation. hist(g1gc.df$UserTime, main="User CPU Time per Collection", xlab="Seconds", ylab="Frequency") Box plot to identify outliers When the initial data is viewed with a box plot, you can see the one crazy outlier in the real time per GC. Save this data point for future analysis and drop the outlier so that it’s not throwing off our statistics. Now the box plot shows many outliers, which will be examined later, using times series analysis. Notice that the scale of the x-axis changes drastically once the crazy outlier is removed. par(mfrow=c(2,1)) boxplot(g1gc.df$UserTime,g1gc.df$SysTime,g1gc.df$RealTime, main="Box Plot of Time per GC\n(dominated by a crazy outlier)", names=c("usr","sys","elapsed"), xlab="Seconds per GC", ylab="Time (Seconds)", horizontal = TRUE, outcol="red") crazy.outlier.df=g1gc.df[g1gc.df$RealTime > 400,] g1gc.df=g1gc.df[g1gc.df$RealTime < 400,] boxplot(g1gc.df$UserTime,g1gc.df$SysTime,g1gc.df$RealTime, main="Box Plot of Time per GC\n(crazy outlier excluded)", names=c("usr","sys","elapsed"), xlab="Seconds per GC", ylab="Time (Seconds)", horizontal = TRUE, outcol="red") box(which = "outer", lty = "solid") Here is the crazy outlier for future analysis: crazy.outlier.df ## row.names SecondsSinceLaunch IncrementalCount ## 8233 2014-05-12T23:15:43.903-0700: 20741 8316 ## FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ## 8233 112 0.55 0.42 488.1 8381440 8235008 9437184 ## Delta ## 8233 146432 R Time Series Data To analyze the garbage collection as a time series, I’ll use Z’s Ordered Observations (zoo). “zoo is the creator for an S3 class of indexed totally ordered observations which includes irregular time series.” require(zoo) ## Loading required package: zoo ## ## Attaching package: 'zoo' ## ## The following objects are masked from 'package:base': ## ## as.Date, as.Date.numeric head(g1gc.df[,1]) ## [1] "2014-05-12T14:00:32.868-0700:" "2014-05-12T14:00:33.179-0700:" ## [3] "2014-05-12T14:00:33.677-0700:" "2014-05-12T14:00:35.538-0700:" ## [5] "2014-05-12T14:00:37.811-0700:" "2014-05-12T14:00:41.428-0700:" options("digits.secs"=3) times=as.POSIXct( g1gc.df[,1], format="%Y-%m-%dT%H:%M:%OS%z:") g1gc.z = zoo(g1gc.df[,-c(1)], order.by=times) head(g1gc.z) ## SecondsSinceLaunch IncrementalCount FullCount ## 2014-05-12 17:00:32.868 1.161 0 0 ## 2014-05-12 17:00:33.178 1.472 1 0 ## 2014-05-12 17:00:33.677 1.969 2 0 ## 2014-05-12 17:00:35.538 3.830 3 0 ## 2014-05-12 17:00:37.811 6.103 4 0 ## 2014-05-12 17:00:41.427 9.720 5 0 ## UserTime SysTime RealTime BeforeSize AfterSize ## 2014-05-12 17:00:32.868 0.11 0.04 0.02 8192 1400 ## 2014-05-12 17:00:33.178 0.05 0.01 0.02 5496 1672 ## 2014-05-12 17:00:33.677 0.04 0.01 0.01 5768 2557 ## 2014-05-12 17:00:35.538 0.21 0.05 0.04 22528 4907 ## 2014-05-12 17:00:37.811 0.08 0.01 0.02 24576 7072 ## 2014-05-12 17:00:41.427 0.26 0.06 0.04 43008 14336 ## TotalSize Delta ## 2014-05-12 17:00:32.868 9437184 6792 ## 2014-05-12 17:00:33.178 9437184 3824 ## 2014-05-12 17:00:33.677 9437184 3211 ## 2014-05-12 17:00:35.538 9437184 17621 ## 2014-05-12 17:00:37.811 9437184 17504 ## 2014-05-12 17:00:41.427 9437184 28672 Example of Two Benchmark Runs in One Log File The data in the following graph is from a different log file, not the one of primary interest to this article. I’m including this image because it is an example of idle periods followed by busy periods. It would be uninteresting to average the rate of garbage collection over the entire log file period. More interesting would be the rate of garbage collect in the two busy periods. Are they the same or different? Your production data may be similar, for example, bursts when employees return from lunch and idle times on weekend evenings, etc. Once the data is in an R Time Series, you can analyze isolated time windows. Clipping the Time Series data Flashing back to our test case… Viewing the data as a time series is interesting. You can see that the work intensive time period is between 9:00 PM and 3:00 AM. Lets clip the data to the interesting period:     par(mfrow=c(2,1)) plot(g1gc.z$UserTime, type="h", main="User Time per GC\nTime: Complete Log File", xlab="Time of Day", ylab="CPU Seconds per GC", col="#1b9e77") clipped.g1gc.z=window(g1gc.z, start=as.POSIXct("2014-05-12 21:00:00"), end=as.POSIXct("2014-05-13 03:00:00")) plot(clipped.g1gc.z$UserTime, type="h", main="User Time per GC\nTime: Limited to Benchmark Execution", xlab="Time of Day", ylab="CPU Seconds per GC", col="#1b9e77") box(which = "outer", lty = "solid") Cumulative Incremental and Full GC count Here is the cumulative incremental and full GC count. When the line is very steep, it indicates that the GCs are repeating very quickly. Notice that the scale on the Y axis is different for full vs. incremental. plot(clipped.g1gc.z[,c(2:3)], main="Cumulative Incremental and Full GC count", xlab="Time of Day", col="#1b9e77") GC Analysis of Benchmark Execution using Time Series data In the following series of 3 graphs: The “After Size” show the amount of heap space in use after each garbage collection. Many Java objects are still referenced, i.e. alive, during each garbage collection. This may indicate that the application has a memory leak, or may indicate that the application has a very large memory footprint. Typically, an application's memory footprint plateau's in the early stage of execution. One would expect this graph to have a flat top. The steep decline in the heap space may indicate that the application crashed after 2:00. The second graph shows that the outliers in real execution time, discussed above, occur near 2:00. when the Java heap seems to be quite full. The third graph shows that Full GCs are infrequent during the first few hours of execution. The rate of Full GC's, (the slope of the cummulative Full GC line), changes near midnight.   plot(clipped.g1gc.z[,c("AfterSize","RealTime","FullCount")], xlab="Time of Day", col=c("#1b9e77","red","#1b9e77")) GC Analysis of heap recovered Each GC trace includes the amount of heap space in use before and after the individual GC event. During garbage coolection, unreferenced objects are identified, the space holding the unreferenced objects is freed, and thus, the difference in before and after usage indicates how much space has been freed. The following box plot and bar chart both demonstrate the same point - the amount of heap space freed per garbage colloection is surprisingly low. par(mfrow=c(2,1)) boxplot(as.vector(clipped.g1gc.z$Delta), main="Amount of Heap Recovered per GC Pass", xlab="Size in KB", horizontal = TRUE, col="red") hist(as.vector(clipped.g1gc.z$Delta), main="Amount of Heap Recovered per GC Pass", xlab="Size in KB", breaks=100, col="red") box(which = "outer", lty = "solid") This graph is the most interesting. The dark blue area shows how much heap is occupied by referenced Java objects. This represents memory that holds live data. The red fringe at the top shows how much data was recovered after each garbage collection. barplot(clipped.g1gc.z[,c("AfterSize","Delta")], col=c("#7570b3","#e7298a"), xlab="Time of Day", border=NA) legend("topleft", c("Live Objects","Heap Recovered on GC"), fill=c("#7570b3","#e7298a")) box(which = "outer", lty = "solid") When I discuss the data in the log files with the customer, I will ask for an explaination for the large amount of referenced data resident in the Java heap. There are two are posibilities: There is a memory leak and the amount of space required to hold referenced objects will continue to grow, limited only by the maximum heap size. After the maximum heap size is reached, the JVM will throw an “Out of Memory” exception every time that the application tries to allocate a new object. If this is the case, the aplication needs to be debugged to identify why old objects are referenced when they are no longer needed. The application has a legitimate requirement to keep a large amount of data in memory. The customer may want to further increase the maximum heap size. Another possible solution would be to partition the application across multiple cluster nodes, where each node has responsibility for managing a unique subset of the data. Conclusion In conclusion, R is a very powerful tool for the analysis of Java garbage collection log files. The primary difficulty is data cleansing so that information can be read into an R data frame. Once the data has been read into R, a rich set of tools may be used for thorough evaluation.

  Read the article

• Zend/PHP: Problem uploading/downloading file to/from MySQL's BLOB field.

  - by NAVEED

  I am uploading file(any type) like this: (It is uploading content of file in blob field of mysql) $organizationModel = new Model_Organization_Object( organizationId ); $myFile = file_get_contents( '../path/to/my/file/filename.ext' ); $organizationModel->setOrganizationProfile( $myFile ); $organizationModel->save(); Now I want to get that file from database and want to download. I doing this in controller's action: (I am aspecting pdf file here therefore it is hardcoded below. But in future I want to download any file from blob field) $organizationModel = new Model_Organization_Object( $organizationId ); $content = $organizationModel->getOrganizationProfile(); header('Content-Type: application/octet-stream'); header("Content-Length: " . strlen($content) ); header('Content-Disposition: attachment; filename=orgProfile.pdf'); $this->view->organizationProfile = $content; Now in view file I am doing this: echo $this-organizationProfile; But above download process print(echo) the content of file in firbug and does not download file in orignal format. My echo output in firebug is like this: %PDF-1.3 %???? 84 0 obj << /Linearized 1 /O 86 /H [ 541 212 ] /L 958398 /E 11238 /N 27 /T 956600 >> endobj xref 84 7 0000000016 00000 n 0000000486 00000 n 0000000753 00000 n 0000000982 00000 n 0000001102 00000 n 0000000541 00000 n 0000000732 00000 n trailer << /Size 91 /Info 83 0 R /Root 85 0 R /Prev 956590 /ID[<0a8d7035bf08791da591e8cae39b8c49><0a8d7035bf08791da591e8cae39b8c49>] >> startxref 0 %%EOF 85 0 obj << /Type /Catalog /Pages 82 0 R >> endobj 89 0 obj << /S 151 /Filter /FlateDecode /Length 90 0 R >> stream H?b```f``?e`b`?f`@\0?.????\\I~aV$?X??dO????bA?Az?lv1o#?{-????1+??G?????N`?b? >?-?? \0\0D40 endstream endobj 90 0 obj 106 endobj 86 0 obj << /Type /Page /Contents 87 0 R /Parent 79 0 R /Resources << /XObject << /img0 88 0 R >> /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] >> /MediaBox [ 0 0 612 792 ] /CropBox [ 0 0 612 792 ] /Rotate 0 >> endobj 87 0 obj << /Filter /FlateDecode /Length 46 >> stream x?+T05???P0\0Bs#SC=S3c3??\\???t?|?@.\0??? endstream endobj 88 0 obj << /Filter /FlateDecode /Type /XObject /Length 8926 /BitsPerComponent 8 /Height 1122 /ColorSpace [ /Indexed /DeviceRGB 255 (\0\0\0JJJkkk{{{????????????????????????????????????\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\\ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0) ] /Subtype /Image /Width 793 >> stream x???v??\0?bF???mf?\\3??k?~? ?7uj??\\\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0~??/\0~??/\0~?&|??tp?pKS ????Fc???!?Q~?72?&???>???]?$?KUo????9?Tx??E8U}?????? _#=??6 Q{????v?T|s?>\\??w??.|??8?7Q????o.?o????????G??x??|?Is:??????oN>4???jJ?F?v? ? V?q<???P?????I>?.|?iT? ???Ç?Q?m????G?8c`????a`<?.|??~`????OG!?x7j??K*]??S?1??_??1\'?D?????0??\"?w\\?e?????<F:4????E-??Fa????O?v????9??_ m???P??8iuTr?i?FX?????<C? ????t:?(0??I>?2`????.???:??pv:???A??<$M??????e9??\\c???.0???t?kum?K;??<???\\@?]f/?h??m_???g???l?8&??*??2?-??Ew?4[j?v?(?????p?T???M--?8 cb??]?h??pN???kt?J$?m???X???5Cr?]?Jm?VP?X?Ð!? ?$???-?PM??O]??,?h???r=???qV}?p*?c?uq??t??????R6v??l8?I?e?9 {s\\K _?CN?^??W?8%p\']?2U?D{???Z?EB?*?d?va1^??Z\"?7?t]?TL?^??d???.|?4?q?2?&2??S{(??G?vNi4?D?K?)_^?]???D]DK???j?9????OQ?]???us?n?T4?om?P??E?|?t??w?????c?7>!]?\"}$??:??<????[9?C??Wi?u?su#9?\0?t?u=??=w??Q??A??.?dyb vN?N\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0??U/????<??v????S?A\\??qkm? !???&??J????!,??+?w;????{?!D???K×%5?E???????|n?FT*,? ?? ?j#Q??uT~r:}\\?_?????v???8Q?&? ???T?S?I\"?(>Y??????}H??aj?3??u?h?T?X?Z?-~??c\'P?^??d?????????]_???z??O]?q?????7??|?mN%??????T?????o???sUzT???m?v8?q? ??e]?wS???C~Ta???.??[%!??????2x]n~?7??????6.????K??;c }????r?)V?? u???*?7?$c\\???m?~???r??)U{?????????R?? ??D1L_????WUog?>??/?????????~?%???M???}\'?? ;???y??K`?????O?,??????<?,0???;3 #??m?v???aZ=?N?u?J`??dwnm;??.??k?n?K1-M?7????H&??????s?C??? ?}Z?1????c?(0?q?_1??7?%???G7U/??h9I??????S?Q??4nc?Lq??H6??;??c(/O??2???-?*_????%?I?/??I?o????ô?k??<????q??\'??]v?\"??+????,????qxgk?\\\\?6???7??Y???.G???Y??8??.??*???M_??J?hu1????z??W?o_??F?/???s?:?Y~??>0?g\\E?l?K5e???&L?/????k$????{?:\\>??Fs?-??l?>c??o?????9?V+?2;??}q?4 ?zS?|u?A`dK???n~?s???K?hiY?j??#p???S?M\\???0P2?\\*?m+?L5Er????[W?>9|???i?????}`Nmc:Qv??]&|?_????fx???????Ns~w??to????K?M???uN????0J6q1??u(b?M?_?????7?]?m?\':????S@???4?????\\??@~Mn?????|}?9?F6_Vr ??7??{?_??_????Y?Go?9??f1????E?|?Ucd? ????????t7k?? }??:??n?M?_????#?M$DG???:Z??y??;g:?|????F?m??e?F*?uJ?C??-?v?%??^?*??????z:l???w?e???9??i?5j???x?~??Ao???a?x?{?UL??? ??#:???\'^?????W??f;?u???ejq¯?u[?2K8??e?>/?ug?@S??L???? ??u0uI~?z?YYV???[[O?T??-Y?u?j?M?_???n&??7O?f??s??z`.`?,W??#?l??n???s??\'?????=??&#?z?M7_????s???x??y? ??u?p?G???0?e?G????8]{??N?1}??}~Q?[)?XF??_??*? p7iQ????M?(?l????????????f??6????*??U;@~\\k?i??w_??*?#???^?j?\\?L??/?}?Y?[??V??t~?w?n??a???m?O?(.?n;??ji:??W?ZnQ[9?n=?^??sE9??;?.??u3\"???<?L??y8?<H???g??u??\\?q???71p?U??}???f`?Y??m3b*C?t{?SX??7m<??6??8K??[Qs??&_??(M??:?Z???W?????W? ??4d??4?A????lw?e?d?>? ?pCV??h?SS?Z?T??4?N?,?? ?8=-?%???4?p?a??~??R?L??=J??j}??"??,?(?x?????????o?ï??t??X7???~jQ?aK???Z*YL????X??/?m?ot?9&s0???O5??j=?7sb?l?Mh???y?}Q\\4?MM?i5&?Yf??hS??N????\'?\0?????i?9??G?$??R?A1[??Y?t??4b?}????u??3?Y??Il????{??[u??f??q???Z_;??|*?t?uTO??}b?a?0>????>?>w\'P?E??]????6???v?^?,?;?uE?f?;?> yo?eNS@?C???I??????Otf????4I??? ?s????*??G?\'?>?</?=T?CE??5NR?~??%?1?d^V??O??????????e||/b??^ \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0~????+?W8??t????????}????????X?????7??/???\\}LM?????b?#?q];J??[U???(0N??t?????[??_V?!??%????-?7:?m???9??Dau?o]??^????*??,???h?o3f??[%?FW?W!?X#; ?>HC?-?\'/??^??\'?*?)??!?_?e?+TC?7O??I.?[?tN?,??Rs???u???^???????q??S??.?c?UR?????? ????M????FS???A????>?^K?|[?]z~7??7u???7V]L?|??l???]?[e?+u?????{U}???Em??IWbV7????/v?x?zk??F@.??5?G?Í>f??_???Gg?}??tc??&R???n???G-?N]/?w?? ????f?}Ue%?;?~?:????`6(??_???g???`? E~?p06?}#/?G=????;??<$Y???l??m?T??@Y??p?????r??.?H?>\0.Ih??~???!?N/^o? ??&v??R???9?suJ?r???JZg?z?Y?7??^?J??H>{[?vQ????qw?e{{?l????????u]?.Z?xh%7??>?|???b`?K?|I\"?nh?m?????m?z5Qpw??N3???y?)??k??????,?Ws*SJ]????????!?o?Iq3~x??Az{?v]\'?k????k???Dc ?]??l?)L??? I8eG#r?dC??;??/C???l???rm???????e?6?M??fP?4?r??)?!?\\s???{??!cN??h??>?? ??o>??m?dO=&<??P??]=]???n?v??y?l??\"?K??????rF?I???)Z??]n?J??N?w???S/S??w???R6}\'u??kN?K`?C/???N??,??o??I?>?S?(??hOV????-]?p?r??0??u?(?,a????/???\"o;???44????P?9K!O]??x?r?}??8?????w?4?|?el7U??l.}|w?- ?=?Lq??e<&??g?/z8??7??:n?????ï??~??_?a???&?7sy???,?3?1??rV???m?????s??C?x50?????g???\\??!??????e?????/Cl?Y???:??jz3??????/?a?]}??\\n???BZ?0?J-+u??????x?=??CC??M??W[??v<???S14?????\\C?Z ??g???q:?u?C?k?vc?K?;??\"Y?t?r]??G?z????w???r??????0??????e?:??/f?*^?W?Q8WsN??9}*?|??~x)?N?=6J?l????M?b??????M45?C?k]??r?u??????r ] Can someone help that how to download file or I am doing something wrong with uploading process. setOrganizationProfile() and getOrganizationProfile() function are created by me which are working fine while storing/getting data to/from database. Thanks

  Read the article

• How to Post Javascript to a MySQL Database

  - by salientanimal

  Hi guys, I have created a form in HTLM and I m trying to post the information in the form to a MySQL database. My form make suse of a dynamic list selection that needs to be captured to the database. However when submtting the form I get the following error Error: Unknown column 'coulmn_name' in 'field list'. Here is my HTML CODE for the form <td height="94"><p align="justify">CALL TRACKER - ADMIN</p></td> Customer Name : E-Mail Address : </tr> <tr> <td width="29%" align="right" valign="middle"><strong>Case Number :</strong></td> <td> <input type="text" name="case_number" width="70%" align="left" valign="middle"> </td> </tr> <tr> <td width="29%" align="right" valign="middle"><strong>MSISDN :</strong></td> <td> <input type="text" name="msisdn" width="70%" align="left" valign="middle"> </td> </tr> <tr> <td width="29%" align="right" valign="middle"> <strong>Route Cause :</strong></td> <td width="71%" align="left" valign="middle"> <select name="route_cause" id="category" onChange="javascript: listboxchange1(this.options[this.selectedIndex].value);"> <!-- <select name="route_cause" id="route_cause"> --> <option value="">Select the Call Reason</option> <option value="Billing Admin">Billing Admin</option> <option value="Customer Care">Customer Care</option> <option value="Insurance">Insurance</option> <option value="Repairs">Repairs</option> <option value="SIM Swap">SIM Swap</option> <option value="UTI">UTI</option> </select> </td> </tr> <tr> <td align="right" valign="middle"> <strong>Call Type/Indexed To :</strong></td> <td align="left" valign="middle"> <script type="text/javascript" language="javascript" name="calltype_indexedto"> <!-- document.write('<select name="subcategory1" onChange="javascript: listboxchange2(this.options[this.selectedIndex].value);"><option value=""></option></select>') --> </script> </td> </tr> <tr> <td align="right" valign="middle"> <strong>Type/TAT :</strong></td> <td align="left" valign="middle"> <script type="text/javascript" language="javascript" name="type_tat"> <!-- document.write('<select name="subcategory2" onChange="javascript: listboxchange3(this.options[this.selectedIndex].value);"><option value=""></option></select>') --> </script> </td> </tr> <tr> <td width="29%" align="right" valign="middle"> <strong>Escalated To :</strong></td> <td width="71%" align="left" valign="middle"> <select name="escalatedto" id="escalated_to"> <option value="">Select the Escalation</option> <option value="Billing Ops">Billing Ops</option> <option value="Resolvers">Resolvers</option> <option value="Finance">Finance</option> <option value="Ressolver">Ressolver</option> <option value="Nudebt">Nudebt</option> <option value="Transunion">Transunion</option> <option value="N/A">N/A</option> </select> </td> </tr> <tr> <td width="29%" align="right" valign="middle"> <strong>Requested By :</strong></td> <td width="71%" align="left" valign="middle"> <select name="requestedby" id="requested_by"> <option value="">UTI Requested By</option> <option value="Billing">Billing</option> <option value="Customer Service">Customer Service</option> <option value="Insurance">Insurance</option> <option value="Management">Management</option> <option value="Repairs">Repairs</option> <option value="Retail Support">Retail Support</option> <option value="Retentions">Retentions</option> <option value="SIM Swap">SIM Swap</option> <option value="WOW">WOW</option> <option value="N/A">N/A</option> </select> </td> </tr> <tr> <td width="29%" align="right" valign="middle"> <strong>Province :</strong></td> <td width="71%" align="left" valign="middle"> <select name="province" id="province"> <option value="">Select the Province</option> <option value="Eastern Cape">Eastern Cape</option> <option value="Gauteng">Gauteng</option> <option value="Kwa-Zulu Natal">Kwa-Zulu Natal</option> <option value="Limpopo">Limpopo</option> <option value="Mpumalanga">Mpumalanga</option> <option value="North West">North West</option> <option value="Northern Cape">Northern Cape</option> <option value="Polokwane">Polokwane</option> <option value="Western Cape">Western Cape</option> <option value="Other">Other</option> </select> </td> </tr> <tr> <td width="29%" align="right" valign="middle"><strong>Comments :</strong></td> <td> <textarea rows ="5" cols="30" name="comments"> </textarea> </td> </tr> <tr> <td> <p> <input type="reset" value="Reset Form"><input type="Submit" value="Submit"> Here is my PHP CODE to write to the Database <?php $con = mysql_connect("hostname" ,"mysqusername" ,"mysqlpassword"); if (!$con) { die('Could not connect: ' . mysql_error()); } mysql_select_db("databasename", $con); $sql="INSERT INTO customer_services_tracker (customer_name ,customer_email_address ,case_number ,msisdn ,route_cause ,calltype_indexedto ,type_tat ,escalatedto ,requestedby ,province ,comments ) VALUES ('$_POST[customer_name]' ,'$_POST[customer_email_address]' ,'$_POST[case_number]' ,'$_POST[msisdn]' ,'$_POST[route_cause]' ,'$_POST[calltype_indexedto]' ,'$_POST[type_tat]' ,'$_POST[escalatedto]' ,'$_POST[requestedby]' ,'$_POST[province]' ,'$_POST[comments]')"; $CatName = $rowCat["Name"]; if (!mysql_query($sql,$con)) { die('Error: ' . mysql_error()); } echo "1 record added"; mysql_close($con) ?>

  Read the article

< Previous Page | 36 37 38 39 40