Search Results

Search found 3363 results on 135 pages for 'postfix operator'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 96/135 | < Previous Page | 92 93 94 95 96 97 98 99 100 101 102 103  | Next Page >

  • What is MySql equivalent of Sql Server Full Text Search?

    - by Nitesh Panchal
    Hello, I have worked with Sql Server in past and used it's very nice feature called Sql Full Text Search. Now i have to work with MySql. Can anybody tell me what is equivalent of Full Text Search in MySql? I am using free edition of MySql and not a commercial one. If not, then what else can we do to mimic Full Text Search and get over the limitations of LIKE operator? Thanks in advance :)

    Read the article

  • Doubt in Conditional inclusion

    - by Philando Gullible
    This is actually extracted from my module (Pre-processor in C) The conditional expression could contain any C operator except for the assignment operators,increment, and decrement operators. I am not sure if I am getting this statement or not since I tried using this and it worked.Also for other manipulation a probable work around would be to simply declare macro or function inside the conditional expression,something like this to be precise. Also I don't understand what is the rationale behind this rule. Could somebody explain? Thanks

    Read the article

  • How to differentiate two tables

    - by Nemat
    I have two tables and I want to get all records from one table that are different from the records in second table. Eg.: if we have four records in the first table like A,B,C,D and three records in the second table thats A,B,C then the answer of query should be D. I have tried "EXCEPT" operator but it doesn't work fine. Kindly help me in writing correct query for the given problem.

    Read the article

  • LINQ display row numbers

    - by timvaines
    I simply want to include a row number against the returned results of my query. I found the following post that describes what I am trying to achieve but gives me an exception http://vaultofthoughts.net/LINQRowNumberColumn.aspx "An expression tree may not contain an assignment operator" In MS SQL I would just use the ROWNUMBER() function, I'm simply looking for the equivalent in LINQ.

    Read the article

  • MongoDB using NOT and AND together

    - by Stankalank
    I'm trying to negate an $and clause with MongoDB and I'm getting a MongoError: invalid operator: $and message back. Basically what I want to achieve is the following: query = { $not: { $and: [{institution_type:'A'}, {type:'C'}] } } Is this possible to express in a mongo query? Here is a sample collection: { "institution_type" : "A", "type" : "C" } { "institution_type" : "A", "type" : "D" } { "institution_type" : "B", "type" : "C" } { "institution_type" : "B", "type" : "D" } What I want to get back is the following: { "institution_type" : "A", "type" : "D" } { "institution_type" : "B", "type" : "C" } { "institution_type" : "B", "type" : "D" }

    Read the article

  • Why doesn't this inner class compile?

    - by Vincenzo
    This is my code: #include <algorithm> class A { void f() { struct CompareMe { bool operator() (int i, int j) { return i < j; } } comp; int a[] = {1, 2, 3, 4}; int found = std::min_element(a[0], a[3], comp); } } Error message: no matching function for call to ‘min_element(int&, int&, A::f()::CompareMe&) What am I doing wrong?

    Read the article

  • Intermediate values in C++

    - by sterh
    Hello. I can not find how to implement a design in C++. In the language of Delphi in case the operator can write the following design: case s[j] of '0'..'9','A'..'Z','a'..'z','_': doSomeThing(); How can i do the same in c++. Attracts me is the construction type 'a' .. 'z' and etc... Thank you

    Read the article

  • Using rowDiffs() to calculate difference in values in matrix

    - by user1723765
    I'm using the rowDiffs() command to calculate the step by step difference in 116 rows in a matrix. I get the following error: Error in r[i1] - r[-length(r):-(length(r) - lag + 1L)] : non-numeric argument to binary operator I have no idea why this is happening. I could take the diff() separately for each row and it would work. Any ideas? Here's the data: https://dl.dropbox.com/u/22681355/data.csv Code: a=rowDiffs(data)

    Read the article

  • AS3 - Can I know if a class implements an interface (or is a subclass of another class) ?

    - by lk
    With this code function someFunction(classParam:Class):Boolean { // how to know if classParam implements some interface? } i.e. Comparing classParam with IEventDispatcher interface: someFunction(EventDispatcher) // returns true someFunction(Object) // returns false I know it can't be done with is operator. But, is there a way to do it? Is there a way to know if a class implements some interface? (or is a subclass of another class?)

    Read the article

  • VC++ 6.0 application crashing inside CString::Format when %d is given.

    - by viswanathan
    A VC++ 6.0 application is crashing when doing a CString::Format operation with %d format specifier. This does not occur always but occurs when the application memory grows upto 100MB or more. ALso sometimes same crash observed when a CString copy is done. The call stack would look like this mfc42u!CFixedAlloc::Alloc+82 mfc42u!CString::AllocBuffer+3f 00000038 00000038 005b5b64 mfc42u!CString::AllocBeforeWrite+31 00000038 0a5bfdbc 005b5b64 mfc42u!CString::AssignCopy+13 00000038 057cb83f 0a5bfe90 mfc42u!CString::operator=+4b and this throws an access violation exception.

    Read the article

  • Using R to Analyze G1GC Log Files

    - by user12620111
    Using R to Analyze G1GC Log Files body, td { font-family: sans-serif; background-color: white; font-size: 12px; margin: 8px; } tt, code, pre { font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; } h1 { font-size:2.2em; } h2 { font-size:1.8em; } h3 { font-size:1.4em; } h4 { font-size:1.0em; } h5 { font-size:0.9em; } h6 { font-size:0.8em; } a:visited { color: rgb(50%, 0%, 50%); } pre { margin-top: 0; max-width: 95%; border: 1px solid #ccc; white-space: pre-wrap; } pre code { display: block; padding: 0.5em; } code.r, code.cpp { background-color: #F8F8F8; } table, td, th { border: none; } blockquote { color:#666666; margin:0; padding-left: 1em; border-left: 0.5em #EEE solid; } hr { height: 0px; border-bottom: none; border-top-width: thin; border-top-style: dotted; border-top-color: #999999; } @media print { * { background: transparent !important; color: black !important; filter:none !important; -ms-filter: none !important; } body { font-size:12pt; max-width:100%; } a, a:visited { text-decoration: underline; } hr { visibility: hidden; page-break-before: always; } pre, blockquote { padding-right: 1em; page-break-inside: avoid; } tr, img { page-break-inside: avoid; } img { max-width: 100% !important; } @page :left { margin: 15mm 20mm 15mm 10mm; } @page :right { margin: 15mm 10mm 15mm 20mm; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3 { page-break-after: avoid; } } pre .operator, pre .paren { color: rgb(104, 118, 135) } pre .literal { color: rgb(88, 72, 246) } pre .number { color: rgb(0, 0, 205); } pre .comment { color: rgb(76, 136, 107); } pre .keyword { color: rgb(0, 0, 255); } pre .identifier { color: rgb(0, 0, 0); } pre .string { color: rgb(3, 106, 7); } var hljs=new function(){function m(p){return p.replace(/&/gm,"&").replace(/"}while(y.length||w.length){var v=u().splice(0,1)[0];z+=m(x.substr(q,v.offset-q));q=v.offset;if(v.event=="start"){z+=t(v.node);s.push(v.node)}else{if(v.event=="stop"){var p,r=s.length;do{r--;p=s[r];z+=("")}while(p!=v.node);s.splice(r,1);while(r'+M[0]+""}else{r+=M[0]}O=P.lR.lastIndex;M=P.lR.exec(L)}return r+L.substr(O,L.length-O)}function J(L,M){if(M.sL&&e[M.sL]){var r=d(M.sL,L);x+=r.keyword_count;return r.value}else{return F(L,M)}}function I(M,r){var L=M.cN?'':"";if(M.rB){y+=L;M.buffer=""}else{if(M.eB){y+=m(r)+L;M.buffer=""}else{y+=L;M.buffer=r}}D.push(M);A+=M.r}function G(N,M,Q){var R=D[D.length-1];if(Q){y+=J(R.buffer+N,R);return false}var P=q(M,R);if(P){y+=J(R.buffer+N,R);I(P,M);return P.rB}var L=v(D.length-1,M);if(L){var O=R.cN?"":"";if(R.rE){y+=J(R.buffer+N,R)+O}else{if(R.eE){y+=J(R.buffer+N,R)+O+m(M)}else{y+=J(R.buffer+N+M,R)+O}}while(L1){O=D[D.length-2].cN?"":"";y+=O;L--;D.length--}var r=D[D.length-1];D.length--;D[D.length-1].buffer="";if(r.starts){I(r.starts,"")}return R.rE}if(w(M,R)){throw"Illegal"}}var E=e[B];var D=[E.dM];var A=0;var x=0;var y="";try{var s,u=0;E.dM.buffer="";do{s=p(C,u);var t=G(s[0],s[1],s[2]);u+=s[0].length;if(!t){u+=s[1].length}}while(!s[2]);if(D.length1){throw"Illegal"}return{r:A,keyword_count:x,value:y}}catch(H){if(H=="Illegal"){return{r:0,keyword_count:0,value:m(C)}}else{throw H}}}function g(t){var p={keyword_count:0,r:0,value:m(t)};var r=p;for(var q in e){if(!e.hasOwnProperty(q)){continue}var s=d(q,t);s.language=q;if(s.keyword_count+s.rr.keyword_count+r.r){r=s}if(s.keyword_count+s.rp.keyword_count+p.r){r=p;p=s}}if(r.language){p.second_best=r}return p}function i(r,q,p){if(q){r=r.replace(/^((]+|\t)+)/gm,function(t,w,v,u){return w.replace(/\t/g,q)})}if(p){r=r.replace(/\n/g,"")}return r}function n(t,w,r){var x=h(t,r);var v=a(t);var y,s;if(v){y=d(v,x)}else{return}var q=c(t);if(q.length){s=document.createElement("pre");s.innerHTML=y.value;y.value=k(q,c(s),x)}y.value=i(y.value,w,r);var u=t.className;if(!u.match("(\\s|^)(language-)?"+v+"(\\s|$)")){u=u?(u+" "+v):v}if(/MSIE [678]/.test(navigator.userAgent)&&t.tagName=="CODE"&&t.parentNode.tagName=="PRE"){s=t.parentNode;var p=document.createElement("div");p.innerHTML=""+y.value+"";t=p.firstChild.firstChild;p.firstChild.cN=s.cN;s.parentNode.replaceChild(p.firstChild,s)}else{t.innerHTML=y.value}t.className=u;t.result={language:v,kw:y.keyword_count,re:y.r};if(y.second_best){t.second_best={language:y.second_best.language,kw:y.second_best.keyword_count,re:y.second_best.r}}}function o(){if(o.called){return}o.called=true;var r=document.getElementsByTagName("pre");for(var p=0;p|=||=||=|\\?|\\[|\\{|\\(|\\^|\\^=|\\||\\|=|\\|\\||~";this.ER="(?![\\s\\S])";this.BE={b:"\\\\.",r:0};this.ASM={cN:"string",b:"'",e:"'",i:"\\n",c:[this.BE],r:0};this.QSM={cN:"string",b:'"',e:'"',i:"\\n",c:[this.BE],r:0};this.CLCM={cN:"comment",b:"//",e:"$"};this.CBLCLM={cN:"comment",b:"/\\*",e:"\\*/"};this.HCM={cN:"comment",b:"#",e:"$"};this.NM={cN:"number",b:this.NR,r:0};this.CNM={cN:"number",b:this.CNR,r:0};this.BNM={cN:"number",b:this.BNR,r:0};this.inherit=function(r,s){var p={};for(var q in r){p[q]=r[q]}if(s){for(var q in s){p[q]=s[q]}}return p}}();hljs.LANGUAGES.cpp=function(){var a={keyword:{"false":1,"int":1,"float":1,"while":1,"private":1,"char":1,"catch":1,"export":1,virtual:1,operator:2,sizeof:2,dynamic_cast:2,typedef:2,const_cast:2,"const":1,struct:1,"for":1,static_cast:2,union:1,namespace:1,unsigned:1,"long":1,"throw":1,"volatile":2,"static":1,"protected":1,bool:1,template:1,mutable:1,"if":1,"public":1,friend:2,"do":1,"return":1,"goto":1,auto:1,"void":2,"enum":1,"else":1,"break":1,"new":1,extern:1,using:1,"true":1,"class":1,asm:1,"case":1,typeid:1,"short":1,reinterpret_cast:2,"default":1,"double":1,register:1,explicit:1,signed:1,typename:1,"try":1,"this":1,"switch":1,"continue":1,wchar_t:1,inline:1,"delete":1,alignof:1,char16_t:1,char32_t:1,constexpr:1,decltype:1,noexcept:1,nullptr:1,static_assert:1,thread_local:1,restrict:1,_Bool:1,complex:1},built_in:{std:1,string:1,cin:1,cout:1,cerr:1,clog:1,stringstream:1,istringstream:1,ostringstream:1,auto_ptr:1,deque:1,list:1,queue:1,stack:1,vector:1,map:1,set:1,bitset:1,multiset:1,multimap:1,unordered_set:1,unordered_map:1,unordered_multiset:1,unordered_multimap:1,array:1,shared_ptr:1}};return{dM:{k:a,i:"",k:a,r:10,c:["self"]}]}}}();hljs.LANGUAGES.r={dM:{c:[hljs.HCM,{cN:"number",b:"\\b0[xX][0-9a-fA-F]+[Li]?\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\b\\d+(?:[eE][+\\-]?\\d*)?L\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\b\\d+\\.(?!\\d)(?:i\\b)?",e:hljs.IMMEDIATE_RE,r:1},{cN:"number",b:"\\b\\d+(?:\\.\\d*)?(?:[eE][+\\-]?\\d*)?i?\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\.\\d+(?:[eE][+\\-]?\\d*)?i?\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"keyword",b:"(?:tryCatch|library|setGeneric|setGroupGeneric)\\b",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\.\\.\\.",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\.\\.\\d+(?![\\w.])",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\b(?:function)",e:hljs.IMMEDIATE_RE,r:2},{cN:"keyword",b:"(?:if|in|break|next|repeat|else|for|return|switch|while|try|stop|warning|require|attach|detach|source|setMethod|setClass)\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"literal",b:"(?:NA|NA_integer_|NA_real_|NA_character_|NA_complex_)\\b",e:hljs.IMMEDIATE_RE,r:10},{cN:"literal",b:"(?:NULL|TRUE|FALSE|T|F|Inf|NaN)\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"identifier",b:"[a-zA-Z.][a-zA-Z0-9._]*\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"operator",b:"|=||   Using R to Analyze G1GC Log Files   Using R to Analyze G1GC Log Files Introduction Working in Oracle Platform Integration gives an engineer opportunities to work on a wide array of technologies. My team’s goal is to make Oracle applications run best on the Solaris/SPARC platform. When looking for bottlenecks in a modern applications, one needs to be aware of not only how the CPUs and operating system are executing, but also network, storage, and in some cases, the Java Virtual Machine. I was recently presented with about 1.5 GB of Java Garbage First Garbage Collector log file data. If you’re not familiar with the subject, you might want to review Garbage First Garbage Collector Tuning by Monica Beckwith. The customer had been running Java HotSpot 1.6.0_31 to host a web application server. I was told that the Solaris/SPARC server was running a Java process launched using a commmand line that included the following flags: -d64 -Xms9g -Xmx9g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=80 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintFlagsFinal -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:ParallelGCThreads=8 Several sources on the internet indicate that if I were to print out the 1.5 GB of log files, it would require enough paper to fill the bed of a pick up truck. Of course, it would be fruitless to try to scan the log files by hand. Tools will be required to summarize the contents of the log files. Others have encountered large Java garbage collection log files. There are existing tools to analyze the log files: IBM’s GC toolkit The chewiebug GCViewer gchisto HPjmeter Instead of using one of the other tools listed, I decide to parse the log files with standard Unix tools, and analyze the data with R. Data Cleansing The log files arrived in two different formats. I guess that the difference is that one set of log files was generated using a more verbose option, maybe -XX:+PrintHeapAtGC, and the other set of log files was generated without that option. Format 1 In some of the log files, the log files with the less verbose format, a single trace, i.e. the report of a singe garbage collection event, looks like this: {Heap before GC invocations=12280 (full 61): garbage-first heap total 9437184K, used 7499918K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) region size 4096K, 1 young (4096K), 0 survivors (0K) compacting perm gen total 262144K, used 144077K [0xffffffff40000000, 0xffffffff50000000, 0xffffffff50000000) the space 262144K, 54% used [0xffffffff40000000, 0xffffffff48cb3758, 0xffffffff48cb3800, 0xffffffff50000000) No shared spaces configured. 2014-05-14T07:24:00.988-0700: 60586.353: [GC pause (young) 7324M->7320M(9216M), 0.1567265 secs] Heap after GC invocations=12281 (full 61): garbage-first heap total 9437184K, used 7496533K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) region size 4096K, 0 young (0K), 0 survivors (0K) compacting perm gen total 262144K, used 144077K [0xffffffff40000000, 0xffffffff50000000, 0xffffffff50000000) the space 262144K, 54% used [0xffffffff40000000, 0xffffffff48cb3758, 0xffffffff48cb3800, 0xffffffff50000000) No shared spaces configured. } A simple grep can be used to extract a summary: $ grep "\[ GC pause (young" g1gc.log 2014-05-13T13:24:35.091-0700: 3.109: [GC pause (young) 20M->5029K(9216M), 0.0146328 secs] 2014-05-13T13:24:35.440-0700: 3.459: [GC pause (young) 9125K->6077K(9216M), 0.0086723 secs] 2014-05-13T13:24:37.581-0700: 5.599: [GC pause (young) 25M->8470K(9216M), 0.0203820 secs] 2014-05-13T13:24:42.686-0700: 10.704: [GC pause (young) 44M->15M(9216M), 0.0288848 secs] 2014-05-13T13:24:48.941-0700: 16.958: [GC pause (young) 51M->20M(9216M), 0.0491244 secs] 2014-05-13T13:24:56.049-0700: 24.066: [GC pause (young) 92M->26M(9216M), 0.0525368 secs] 2014-05-13T13:25:34.368-0700: 62.383: [GC pause (young) 602M->68M(9216M), 0.1721173 secs] But that format wasn't easily read into R, so I needed to be a bit more tricky. I used the following Unix command to create a summary file that was easy for R to read. $ echo "SecondsSinceLaunch BeforeSize AfterSize TotalSize RealTime" $ grep "\[GC pause (young" g1gc.log | grep -v mark | sed -e 's/[A-SU-z\(\),]/ /g' -e 's/->/ /' -e 's/: / /g' | more SecondsSinceLaunch BeforeSize AfterSize TotalSize RealTime 2014-05-13T13:24:35.091-0700 3.109 20 5029 9216 0.0146328 2014-05-13T13:24:35.440-0700 3.459 9125 6077 9216 0.0086723 2014-05-13T13:24:37.581-0700 5.599 25 8470 9216 0.0203820 2014-05-13T13:24:42.686-0700 10.704 44 15 9216 0.0288848 2014-05-13T13:24:48.941-0700 16.958 51 20 9216 0.0491244 2014-05-13T13:24:56.049-0700 24.066 92 26 9216 0.0525368 2014-05-13T13:25:34.368-0700 62.383 602 68 9216 0.1721173 Format 2 In some of the log files, the log files with the more verbose format, a single trace, i.e. the report of a singe garbage collection event, was more complicated than Format 1. Here is a text file with an example of a single G1GC trace in the second format. As you can see, it is quite complicated. It is nice that there is so much information available, but the level of detail can be overwhelming. I wrote this awk script (download) to summarize each trace on a single line. #!/usr/bin/env awk -f BEGIN { printf("SecondsSinceLaunch IncrementalCount FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize\n") } ###################### # Save count data from lines that are at the start of each G1GC trace. # Each trace starts out like this: # {Heap before GC invocations=14 (full 0): # garbage-first heap total 9437184K, used 325496K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) ###################### /{Heap.*full/{ gsub ( "\\)" , "" ); nf=split($0,a,"="); split(a[2],b," "); getline; if ( match($0, "first") ) { G1GC=1; IncrementalCount=b[1]; FullCount=substr( b[3], 1, length(b[3])-1 ); } else { G1GC=0; } } ###################### # Pull out time stamps that are in lines with this format: # 2014-05-12T14:02:06.025-0700: 94.312: [GC pause (young), 0.08870154 secs] ###################### /GC pause/ { DateTime=$1; SecondsSinceLaunch=substr($2, 1, length($2)-1); } ###################### # Heap sizes are in lines that look like this: # [ 4842M->4838M(9216M)] ###################### /\[ .*]$/ { gsub ( "\\[" , "" ); gsub ( "\ \]" , "" ); gsub ( "->" , " " ); gsub ( "\\( " , " " ); gsub ( "\ \)" , " " ); split($0,a," "); if ( split(a[1],b,"M") > 1 ) {BeforeSize=b[1]*1024;} if ( split(a[1],b,"K") > 1 ) {BeforeSize=b[1];} if ( split(a[2],b,"M") > 1 ) {AfterSize=b[1]*1024;} if ( split(a[2],b,"K") > 1 ) {AfterSize=b[1];} if ( split(a[3],b,"M") > 1 ) {TotalSize=b[1]*1024;} if ( split(a[3],b,"K") > 1 ) {TotalSize=b[1];} } ###################### # Emit an output line when you find input that looks like this: # [Times: user=1.41 sys=0.08, real=0.24 secs] ###################### /\[Times/ { if (G1GC==1) { gsub ( "," , "" ); split($2,a,"="); UserTime=a[2]; split($3,a,"="); SysTime=a[2]; split($4,a,"="); RealTime=a[2]; print DateTime,SecondsSinceLaunch,IncrementalCount,FullCount,UserTime,SysTime,RealTime,BeforeSize,AfterSize,TotalSize; G1GC=0; } } The resulting summary is about 25X smaller that the original file, but still difficult for a human to digest. SecondsSinceLaunch IncrementalCount FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ... 2014-05-12T18:36:34.669-0700: 3985.744 561 0 0.57 0.06 0.16 1724416 1720320 9437184 2014-05-12T18:36:34.839-0700: 3985.914 562 0 0.51 0.06 0.19 1724416 1720320 9437184 2014-05-12T18:36:35.069-0700: 3986.144 563 0 0.60 0.04 0.27 1724416 1721344 9437184 2014-05-12T18:36:35.354-0700: 3986.429 564 0 0.33 0.04 0.09 1725440 1722368 9437184 2014-05-12T18:36:35.545-0700: 3986.620 565 0 0.58 0.04 0.17 1726464 1722368 9437184 2014-05-12T18:36:35.726-0700: 3986.801 566 0 0.43 0.05 0.12 1726464 1722368 9437184 2014-05-12T18:36:35.856-0700: 3986.930 567 0 0.30 0.04 0.07 1726464 1723392 9437184 2014-05-12T18:36:35.947-0700: 3987.023 568 0 0.61 0.04 0.26 1727488 1723392 9437184 2014-05-12T18:36:36.228-0700: 3987.302 569 0 0.46 0.04 0.16 1731584 1724416 9437184 Reading the Data into R Once the GC log data had been cleansed, either by processing the first format with the shell script, or by processing the second format with the awk script, it was easy to read the data into R. g1gc.df = read.csv("summary.txt", row.names = NULL, stringsAsFactors=FALSE,sep="") str(g1gc.df) ## 'data.frame': 8307 obs. of 10 variables: ## $ row.names : chr "2014-05-12T14:00:32.868-0700:" "2014-05-12T14:00:33.179-0700:" "2014-05-12T14:00:33.677-0700:" "2014-05-12T14:00:35.538-0700:" ... ## $ SecondsSinceLaunch: num 1.16 1.47 1.97 3.83 6.1 ... ## $ IncrementalCount : int 0 1 2 3 4 5 6 7 8 9 ... ## $ FullCount : int 0 0 0 0 0 0 0 0 0 0 ... ## $ UserTime : num 0.11 0.05 0.04 0.21 0.08 0.26 0.31 0.33 0.34 0.56 ... ## $ SysTime : num 0.04 0.01 0.01 0.05 0.01 0.06 0.07 0.06 0.07 0.09 ... ## $ RealTime : num 0.02 0.02 0.01 0.04 0.02 0.04 0.05 0.04 0.04 0.06 ... ## $ BeforeSize : int 8192 5496 5768 22528 24576 43008 34816 53248 55296 93184 ... ## $ AfterSize : int 1400 1672 2557 4907 7072 14336 16384 18432 19456 21504 ... ## $ TotalSize : int 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 ... head(g1gc.df) ## row.names SecondsSinceLaunch IncrementalCount ## 1 2014-05-12T14:00:32.868-0700: 1.161 0 ## 2 2014-05-12T14:00:33.179-0700: 1.472 1 ## 3 2014-05-12T14:00:33.677-0700: 1.969 2 ## 4 2014-05-12T14:00:35.538-0700: 3.830 3 ## 5 2014-05-12T14:00:37.811-0700: 6.103 4 ## 6 2014-05-12T14:00:41.428-0700: 9.720 5 ## FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ## 1 0 0.11 0.04 0.02 8192 1400 9437184 ## 2 0 0.05 0.01 0.02 5496 1672 9437184 ## 3 0 0.04 0.01 0.01 5768 2557 9437184 ## 4 0 0.21 0.05 0.04 22528 4907 9437184 ## 5 0 0.08 0.01 0.02 24576 7072 9437184 ## 6 0 0.26 0.06 0.04 43008 14336 9437184 Basic Statistics Once the data has been read into R, simple statistics are very easy to generate. All of the numbers from high school statistics are available via simple commands. For example, generate a summary of every column: summary(g1gc.df) ## row.names SecondsSinceLaunch IncrementalCount FullCount ## Length:8307 Min. : 1 Min. : 0 Min. : 0.0 ## Class :character 1st Qu.: 9977 1st Qu.:2048 1st Qu.: 0.0 ## Mode :character Median :12855 Median :4136 Median : 12.0 ## Mean :12527 Mean :4156 Mean : 31.6 ## 3rd Qu.:15758 3rd Qu.:6262 3rd Qu.: 61.0 ## Max. :55484 Max. :8391 Max. :113.0 ## UserTime SysTime RealTime BeforeSize ## Min. :0.040 Min. :0.0000 Min. : 0.0 Min. : 5476 ## 1st Qu.:0.470 1st Qu.:0.0300 1st Qu.: 0.1 1st Qu.:5137920 ## Median :0.620 Median :0.0300 Median : 0.1 Median :6574080 ## Mean :0.751 Mean :0.0355 Mean : 0.3 Mean :5841855 ## 3rd Qu.:0.920 3rd Qu.:0.0400 3rd Qu.: 0.2 3rd Qu.:7084032 ## Max. :3.370 Max. :1.5600 Max. :488.1 Max. :8696832 ## AfterSize TotalSize ## Min. : 1380 Min. :9437184 ## 1st Qu.:5002752 1st Qu.:9437184 ## Median :6559744 Median :9437184 ## Mean :5785454 Mean :9437184 ## 3rd Qu.:7054336 3rd Qu.:9437184 ## Max. :8482816 Max. :9437184 Q: What is the total amount of User CPU time spent in garbage collection? sum(g1gc.df$UserTime) ## [1] 6236 As you can see, less than two hours of CPU time was spent in garbage collection. Is that too much? To find the percentage of time spent in garbage collection, divide the number above by total_elapsed_time*CPU_count. In this case, there are a lot of CPU’s and it turns out the the overall amount of CPU time spent in garbage collection isn’t a problem when viewed in isolation. When calculating rates, i.e. events per unit time, you need to ask yourself if the rate is homogenous across the time period in the log file. Does the log file include spikes of high activity that should be separately analyzed? Averaging in data from nights and weekends with data from business hours may alias problems. If you have a reason to suspect that the garbage collection rates include peaks and valleys that need independent analysis, see the “Time Series” section, below. Q: How much garbage is collected on each pass? The amount of heap space that is recovered per GC pass is surprisingly low: At least one collection didn’t recover any data. (“Min.=0”) 25% of the passes recovered 3MB or less. (“1st Qu.=3072”) Half of the GC passes recovered 4MB or less. (“Median=4096”) The average amount recovered was 56MB. (“Mean=56390”) 75% of the passes recovered 36MB or less. (“3rd Qu.=36860”) At least one pass recovered 2GB. (“Max.=2121000”) g1gc.df$Delta = g1gc.df$BeforeSize - g1gc.df$AfterSize summary(g1gc.df$Delta) ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0 3070 4100 56400 36900 2120000 Q: What is the maximum User CPU time for a single collection? The worst garbage collection (“Max.”) is many standard deviations away from the mean. The data appears to be right skewed. summary(g1gc.df$UserTime) ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0.040 0.470 0.620 0.751 0.920 3.370 sd(g1gc.df$UserTime) ## [1] 0.3966 Basic Graphics Once the data is in R, it is trivial to plot the data with formats including dot plots, line charts, bar charts (simple, stacked, grouped), pie charts, boxplots, scatter plots histograms, and kernel density plots. Histogram of User CPU Time per Collection I don't think that this graph requires any explanation. hist(g1gc.df$UserTime, main="User CPU Time per Collection", xlab="Seconds", ylab="Frequency") Box plot to identify outliers When the initial data is viewed with a box plot, you can see the one crazy outlier in the real time per GC. Save this data point for future analysis and drop the outlier so that it’s not throwing off our statistics. Now the box plot shows many outliers, which will be examined later, using times series analysis. Notice that the scale of the x-axis changes drastically once the crazy outlier is removed. par(mfrow=c(2,1)) boxplot(g1gc.df$UserTime,g1gc.df$SysTime,g1gc.df$RealTime, main="Box Plot of Time per GC\n(dominated by a crazy outlier)", names=c("usr","sys","elapsed"), xlab="Seconds per GC", ylab="Time (Seconds)", horizontal = TRUE, outcol="red") crazy.outlier.df=g1gc.df[g1gc.df$RealTime > 400,] g1gc.df=g1gc.df[g1gc.df$RealTime < 400,] boxplot(g1gc.df$UserTime,g1gc.df$SysTime,g1gc.df$RealTime, main="Box Plot of Time per GC\n(crazy outlier excluded)", names=c("usr","sys","elapsed"), xlab="Seconds per GC", ylab="Time (Seconds)", horizontal = TRUE, outcol="red") box(which = "outer", lty = "solid") Here is the crazy outlier for future analysis: crazy.outlier.df ## row.names SecondsSinceLaunch IncrementalCount ## 8233 2014-05-12T23:15:43.903-0700: 20741 8316 ## FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ## 8233 112 0.55 0.42 488.1 8381440 8235008 9437184 ## Delta ## 8233 146432 R Time Series Data To analyze the garbage collection as a time series, I’ll use Z’s Ordered Observations (zoo). “zoo is the creator for an S3 class of indexed totally ordered observations which includes irregular time series.” require(zoo) ## Loading required package: zoo ## ## Attaching package: 'zoo' ## ## The following objects are masked from 'package:base': ## ## as.Date, as.Date.numeric head(g1gc.df[,1]) ## [1] "2014-05-12T14:00:32.868-0700:" "2014-05-12T14:00:33.179-0700:" ## [3] "2014-05-12T14:00:33.677-0700:" "2014-05-12T14:00:35.538-0700:" ## [5] "2014-05-12T14:00:37.811-0700:" "2014-05-12T14:00:41.428-0700:" options("digits.secs"=3) times=as.POSIXct( g1gc.df[,1], format="%Y-%m-%dT%H:%M:%OS%z:") g1gc.z = zoo(g1gc.df[,-c(1)], order.by=times) head(g1gc.z) ## SecondsSinceLaunch IncrementalCount FullCount ## 2014-05-12 17:00:32.868 1.161 0 0 ## 2014-05-12 17:00:33.178 1.472 1 0 ## 2014-05-12 17:00:33.677 1.969 2 0 ## 2014-05-12 17:00:35.538 3.830 3 0 ## 2014-05-12 17:00:37.811 6.103 4 0 ## 2014-05-12 17:00:41.427 9.720 5 0 ## UserTime SysTime RealTime BeforeSize AfterSize ## 2014-05-12 17:00:32.868 0.11 0.04 0.02 8192 1400 ## 2014-05-12 17:00:33.178 0.05 0.01 0.02 5496 1672 ## 2014-05-12 17:00:33.677 0.04 0.01 0.01 5768 2557 ## 2014-05-12 17:00:35.538 0.21 0.05 0.04 22528 4907 ## 2014-05-12 17:00:37.811 0.08 0.01 0.02 24576 7072 ## 2014-05-12 17:00:41.427 0.26 0.06 0.04 43008 14336 ## TotalSize Delta ## 2014-05-12 17:00:32.868 9437184 6792 ## 2014-05-12 17:00:33.178 9437184 3824 ## 2014-05-12 17:00:33.677 9437184 3211 ## 2014-05-12 17:00:35.538 9437184 17621 ## 2014-05-12 17:00:37.811 9437184 17504 ## 2014-05-12 17:00:41.427 9437184 28672 Example of Two Benchmark Runs in One Log File The data in the following graph is from a different log file, not the one of primary interest to this article. I’m including this image because it is an example of idle periods followed by busy periods. It would be uninteresting to average the rate of garbage collection over the entire log file period. More interesting would be the rate of garbage collect in the two busy periods. Are they the same or different? Your production data may be similar, for example, bursts when employees return from lunch and idle times on weekend evenings, etc. Once the data is in an R Time Series, you can analyze isolated time windows. Clipping the Time Series data Flashing back to our test case… Viewing the data as a time series is interesting. You can see that the work intensive time period is between 9:00 PM and 3:00 AM. Lets clip the data to the interesting period:     par(mfrow=c(2,1)) plot(g1gc.z$UserTime, type="h", main="User Time per GC\nTime: Complete Log File", xlab="Time of Day", ylab="CPU Seconds per GC", col="#1b9e77") clipped.g1gc.z=window(g1gc.z, start=as.POSIXct("2014-05-12 21:00:00"), end=as.POSIXct("2014-05-13 03:00:00")) plot(clipped.g1gc.z$UserTime, type="h", main="User Time per GC\nTime: Limited to Benchmark Execution", xlab="Time of Day", ylab="CPU Seconds per GC", col="#1b9e77") box(which = "outer", lty = "solid") Cumulative Incremental and Full GC count Here is the cumulative incremental and full GC count. When the line is very steep, it indicates that the GCs are repeating very quickly. Notice that the scale on the Y axis is different for full vs. incremental. plot(clipped.g1gc.z[,c(2:3)], main="Cumulative Incremental and Full GC count", xlab="Time of Day", col="#1b9e77") GC Analysis of Benchmark Execution using Time Series data In the following series of 3 graphs: The “After Size” show the amount of heap space in use after each garbage collection. Many Java objects are still referenced, i.e. alive, during each garbage collection. This may indicate that the application has a memory leak, or may indicate that the application has a very large memory footprint. Typically, an application's memory footprint plateau's in the early stage of execution. One would expect this graph to have a flat top. The steep decline in the heap space may indicate that the application crashed after 2:00. The second graph shows that the outliers in real execution time, discussed above, occur near 2:00. when the Java heap seems to be quite full. The third graph shows that Full GCs are infrequent during the first few hours of execution. The rate of Full GC's, (the slope of the cummulative Full GC line), changes near midnight.   plot(clipped.g1gc.z[,c("AfterSize","RealTime","FullCount")], xlab="Time of Day", col=c("#1b9e77","red","#1b9e77")) GC Analysis of heap recovered Each GC trace includes the amount of heap space in use before and after the individual GC event. During garbage coolection, unreferenced objects are identified, the space holding the unreferenced objects is freed, and thus, the difference in before and after usage indicates how much space has been freed. The following box plot and bar chart both demonstrate the same point - the amount of heap space freed per garbage colloection is surprisingly low. par(mfrow=c(2,1)) boxplot(as.vector(clipped.g1gc.z$Delta), main="Amount of Heap Recovered per GC Pass", xlab="Size in KB", horizontal = TRUE, col="red") hist(as.vector(clipped.g1gc.z$Delta), main="Amount of Heap Recovered per GC Pass", xlab="Size in KB", breaks=100, col="red") box(which = "outer", lty = "solid") This graph is the most interesting. The dark blue area shows how much heap is occupied by referenced Java objects. This represents memory that holds live data. The red fringe at the top shows how much data was recovered after each garbage collection. barplot(clipped.g1gc.z[,c("AfterSize","Delta")], col=c("#7570b3","#e7298a"), xlab="Time of Day", border=NA) legend("topleft", c("Live Objects","Heap Recovered on GC"), fill=c("#7570b3","#e7298a")) box(which = "outer", lty = "solid") When I discuss the data in the log files with the customer, I will ask for an explaination for the large amount of referenced data resident in the Java heap. There are two are posibilities: There is a memory leak and the amount of space required to hold referenced objects will continue to grow, limited only by the maximum heap size. After the maximum heap size is reached, the JVM will throw an “Out of Memory” exception every time that the application tries to allocate a new object. If this is the case, the aplication needs to be debugged to identify why old objects are referenced when they are no longer needed. The application has a legitimate requirement to keep a large amount of data in memory. The customer may want to further increase the maximum heap size. Another possible solution would be to partition the application across multiple cluster nodes, where each node has responsibility for managing a unique subset of the data. Conclusion In conclusion, R is a very powerful tool for the analysis of Java garbage collection log files. The primary difficulty is data cleansing so that information can be read into an R data frame. Once the data has been read into R, a rich set of tools may be used for thorough evaluation.

    Read the article

  • High Linux loads on low CPU/memory usage

    - by user13323
    Hi. I have quite strange situation, where my CentOS 5.5 box loads are high, but the CPU and memory used are pretty low: top - 20:41:38 up 42 days, 6:14, 2 users, load average: 19.79, 21.25, 18.87 Tasks: 254 total, 1 running, 253 sleeping, 0 stopped, 0 zombie Cpu(s): 3.8%us, 0.3%sy, 0.1%ni, 95.0%id, 0.6%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 4035284k total, 4008084k used, 27200k free, 38748k buffers Swap: 4208928k total, 242576k used, 3966352k free, 1465008k cached free -mt total used free shared buffers cached Mem: 3940 3910 29 0 37 1427 -/+ buffers/cache: 2445 1495 Swap: 4110 236 3873 Total: 8050 4147 3903 Iostat also shows good results: avg-cpu: %user %nice %system %iowait %steal %idle 3.83 0.13 0.41 0.58 0.00 95.05 Here is the ps aux output: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 10348 80 ? Ss 2010 2:11 init [3] root 2 0.0 0.0 0 0 ? S< 2010 0:00 [migration/0] root 3 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/0] root 5 0.0 0.0 0 0 ? S< 2010 0:02 [migration/1] root 6 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/1] root 8 0.0 0.0 0 0 ? S< 2010 0:02 [migration/2] root 9 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/2] root 10 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/2] root 11 0.0 0.0 0 0 ? S< 2010 0:02 [migration/3] root 12 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/3] root 13 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/3] root 14 0.0 0.0 0 0 ? S< 2010 0:03 [migration/4] root 15 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/4] root 16 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/4] root 17 0.0 0.0 0 0 ? S< 2010 0:01 [migration/5] root 18 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/5] root 19 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/5] root 20 0.0 0.0 0 0 ? S< 2010 0:11 [migration/6] root 21 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/6] root 22 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/6] root 23 0.0 0.0 0 0 ? S< 2010 0:01 [migration/7] root 24 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/7] root 25 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/7] root 26 0.0 0.0 0 0 ? S< 2010 0:00 [migration/8] root 27 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/8] root 28 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/8] root 29 0.0 0.0 0 0 ? S< 2010 0:00 [migration/9] root 30 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/9] root 31 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/9] root 32 0.0 0.0 0 0 ? S< 2010 0:08 [migration/10] root 33 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/10] root 34 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/10] root 35 0.0 0.0 0 0 ? S< 2010 0:05 [migration/11] root 36 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/11] root 37 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/11] root 38 0.0 0.0 0 0 ? S< 2010 0:02 [migration/12] root 39 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/12] root 40 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/12] root 41 0.0 0.0 0 0 ? S< 2010 0:14 [migration/13] root 42 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/13] root 43 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/13] root 44 0.0 0.0 0 0 ? S< 2010 0:04 [migration/14] root 45 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/14] root 46 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/14] root 47 0.0 0.0 0 0 ? S< 2010 0:01 [migration/15] root 48 0.0 0.0 0 0 ? SN 2010 0:00 [ksoftirqd/15] root 49 0.0 0.0 0 0 ? S< 2010 0:00 [watchdog/15] root 50 0.0 0.0 0 0 ? S< 2010 0:00 [events/0] root 51 0.0 0.0 0 0 ? S< 2010 0:00 [events/1] root 52 0.0 0.0 0 0 ? S< 2010 0:00 [events/2] root 53 0.0 0.0 0 0 ? S< 2010 0:00 [events/3] root 54 0.0 0.0 0 0 ? S< 2010 0:00 [events/4] root 55 0.0 0.0 0 0 ? S< 2010 0:00 [events/5] root 56 0.0 0.0 0 0 ? S< 2010 0:00 [events/6] root 57 0.0 0.0 0 0 ? S< 2010 0:00 [events/7] root 58 0.0 0.0 0 0 ? S< 2010 0:00 [events/8] root 59 0.0 0.0 0 0 ? S< 2010 0:00 [events/9] root 60 0.0 0.0 0 0 ? S< 2010 0:00 [events/10] root 61 0.0 0.0 0 0 ? S< 2010 0:00 [events/11] root 62 0.0 0.0 0 0 ? S< 2010 0:00 [events/12] root 63 0.0 0.0 0 0 ? S< 2010 0:00 [events/13] root 64 0.0 0.0 0 0 ? S< 2010 0:00 [events/14] root 65 0.0 0.0 0 0 ? S< 2010 0:00 [events/15] root 66 0.0 0.0 0 0 ? S< 2010 0:00 [khelper] root 107 0.0 0.0 0 0 ? S< 2010 0:00 [kthread] root 126 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/0] root 127 0.0 0.0 0 0 ? S< 2010 0:03 [kblockd/1] root 128 0.0 0.0 0 0 ? S< 2010 0:01 [kblockd/2] root 129 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/3] root 130 0.0 0.0 0 0 ? S< 2010 0:05 [kblockd/4] root 131 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/5] root 132 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/6] root 133 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/7] root 134 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/8] root 135 0.0 0.0 0 0 ? S< 2010 0:02 [kblockd/9] root 136 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/10] root 137 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/11] root 138 0.0 0.0 0 0 ? S< 2010 0:04 [kblockd/12] root 139 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/13] root 140 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/14] root 141 0.0 0.0 0 0 ? S< 2010 0:00 [kblockd/15] root 142 0.0 0.0 0 0 ? S< 2010 0:00 [kacpid] root 281 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/0] root 282 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/1] root 283 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/2] root 284 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/3] root 285 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/4] root 286 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/5] root 287 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/6] root 288 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/7] root 289 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/8] root 290 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/9] root 291 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/10] root 292 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/11] root 293 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/12] root 294 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/13] root 295 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/14] root 296 0.0 0.0 0 0 ? S< 2010 0:00 [cqueue/15] root 299 0.0 0.0 0 0 ? S< 2010 0:00 [khubd] root 301 0.0 0.0 0 0 ? S< 2010 0:00 [kseriod] root 490 0.0 0.0 0 0 ? S 2010 0:00 [khungtaskd] root 493 0.1 0.0 0 0 ? S< 2010 94:48 [kswapd1] root 494 0.0 0.0 0 0 ? S< 2010 0:00 [aio/0] root 495 0.0 0.0 0 0 ? S< 2010 0:00 [aio/1] root 496 0.0 0.0 0 0 ? S< 2010 0:00 [aio/2] root 497 0.0 0.0 0 0 ? S< 2010 0:00 [aio/3] root 498 0.0 0.0 0 0 ? S< 2010 0:00 [aio/4] root 499 0.0 0.0 0 0 ? S< 2010 0:00 [aio/5] root 500 0.0 0.0 0 0 ? S< 2010 0:00 [aio/6] root 501 0.0 0.0 0 0 ? S< 2010 0:00 [aio/7] root 502 0.0 0.0 0 0 ? S< 2010 0:00 [aio/8] root 503 0.0 0.0 0 0 ? S< 2010 0:00 [aio/9] root 504 0.0 0.0 0 0 ? S< 2010 0:00 [aio/10] root 505 0.0 0.0 0 0 ? S< 2010 0:00 [aio/11] root 506 0.0 0.0 0 0 ? S< 2010 0:00 [aio/12] root 507 0.0 0.0 0 0 ? S< 2010 0:00 [aio/13] root 508 0.0 0.0 0 0 ? S< 2010 0:00 [aio/14] root 509 0.0 0.0 0 0 ? S< 2010 0:00 [aio/15] root 665 0.0 0.0 0 0 ? S< 2010 0:00 [kpsmoused] root 808 0.0 0.0 0 0 ? S< 2010 0:00 [ata/0] root 809 0.0 0.0 0 0 ? S< 2010 0:00 [ata/1] root 810 0.0 0.0 0 0 ? S< 2010 0:00 [ata/2] root 811 0.0 0.0 0 0 ? S< 2010 0:00 [ata/3] root 812 0.0 0.0 0 0 ? S< 2010 0:00 [ata/4] root 813 0.0 0.0 0 0 ? S< 2010 0:00 [ata/5] root 814 0.0 0.0 0 0 ? S< 2010 0:00 [ata/6] root 815 0.0 0.0 0 0 ? S< 2010 0:00 [ata/7] root 816 0.0 0.0 0 0 ? S< 2010 0:00 [ata/8] root 817 0.0 0.0 0 0 ? S< 2010 0:00 [ata/9] root 818 0.0 0.0 0 0 ? S< 2010 0:00 [ata/10] root 819 0.0 0.0 0 0 ? S< 2010 0:00 [ata/11] root 820 0.0 0.0 0 0 ? S< 2010 0:00 [ata/12] root 821 0.0 0.0 0 0 ? S< 2010 0:00 [ata/13] root 822 0.0 0.0 0 0 ? S< 2010 0:00 [ata/14] root 823 0.0 0.0 0 0 ? S< 2010 0:00 [ata/15] root 824 0.0 0.0 0 0 ? S< 2010 0:00 [ata_aux] root 842 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_0] root 843 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_1] root 844 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_2] root 845 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_3] root 846 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_4] root 847 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_5] root 882 0.0 0.0 0 0 ? S< 2010 0:00 [kstriped] root 951 0.0 0.0 0 0 ? S< 2010 4:24 [kjournald] root 976 0.0 0.0 0 0 ? S< 2010 0:00 [kauditd] postfix 990 0.0 0.0 54208 2284 ? S 21:19 0:00 pickup -l -t fifo -u root 1013 0.0 0.0 12676 8 ? S<s 2010 0:00 /sbin/udevd -d root 1326 0.0 0.0 90900 3400 ? Ss 14:53 0:00 sshd: root@notty root 1410 0.0 0.0 53972 2108 ? Ss 14:53 0:00 /usr/libexec/openssh/sftp-server root 2690 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/0] root 2691 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/1] root 2692 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/2] root 2693 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/3] root 2694 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/4] root 2695 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/5] root 2696 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/6] root 2697 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/7] root 2698 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/8] root 2699 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/9] root 2700 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/10] root 2701 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/11] root 2702 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/12] root 2703 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/13] root 2704 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/14] root 2705 0.0 0.0 0 0 ? S< 2010 0:00 [kmpathd/15] root 2706 0.0 0.0 0 0 ? S< 2010 0:00 [kmpath_handlerd] root 2755 0.0 0.0 0 0 ? S< 2010 4:35 [kjournald] root 2757 0.0 0.0 0 0 ? S< 2010 3:38 [kjournald] root 2759 0.0 0.0 0 0 ? S< 2010 4:10 [kjournald] root 2761 0.0 0.0 0 0 ? S< 2010 4:26 [kjournald] root 2763 0.0 0.0 0 0 ? S< 2010 3:15 [kjournald] root 2765 0.0 0.0 0 0 ? S< 2010 3:04 [kjournald] root 2767 0.0 0.0 0 0 ? S< 2010 3:02 [kjournald] root 2769 0.0 0.0 0 0 ? S< 2010 2:58 [kjournald] root 2771 0.0 0.0 0 0 ? S< 2010 0:00 [kjournald] root 3340 0.0 0.0 5908 356 ? Ss 2010 2:48 syslogd -m 0 root 3343 0.0 0.0 3804 212 ? Ss 2010 0:03 klogd -x root 3430 0.0 0.0 0 0 ? S< 2010 0:50 [kondemand/0] root 3431 0.0 0.0 0 0 ? S< 2010 0:54 [kondemand/1] root 3432 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/2] root 3433 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/3] root 3434 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/4] root 3435 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/5] root 3436 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/6] root 3437 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/7] root 3438 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/8] root 3439 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/9] root 3440 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/10] root 3441 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/11] root 3442 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/12] root 3443 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/13] root 3444 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/14] root 3445 0.0 0.0 0 0 ? S< 2010 0:00 [kondemand/15] root 3461 0.0 0.0 10760 284 ? Ss 2010 3:44 irqbalance rpc 3481 0.0 0.0 8052 4 ? Ss 2010 0:00 portmap root 3526 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/0] root 3527 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/1] root 3528 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/2] root 3529 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/3] root 3530 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/4] root 3531 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/5] root 3532 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/6] root 3533 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/7] root 3534 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/8] root 3535 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/9] root 3536 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/10] root 3537 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/11] root 3538 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/12] root 3539 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/13] root 3540 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/14] root 3541 0.0 0.0 0 0 ? S< 2010 0:00 [rpciod/15] root 3563 0.0 0.0 10160 8 ? Ss 2010 0:00 rpc.statd root 3595 0.0 0.0 55180 4 ? Ss 2010 0:00 rpc.idmapd dbus 3618 0.0 0.0 21256 28 ? Ss 2010 0:00 dbus-daemon --system root 3649 0.2 0.4 563084 18796 ? S<sl 2010 179:03 mfsmount /mnt/mfs -o rw,mfsmaster=web1.ovs.local root 3702 0.0 0.0 3800 8 ? Ss 2010 0:00 /usr/sbin/acpid 68 3715 0.0 0.0 31312 816 ? Ss 2010 3:14 hald root 3716 0.0 0.0 21692 28 ? S 2010 0:00 hald-runner 68 3726 0.0 0.0 12324 8 ? S 2010 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket 68 3730 0.0 0.0 12324 8 ? S 2010 0:00 hald-addon-keyboard: listening on /dev/input/event0 root 3773 0.0 0.0 62608 332 ? Ss 2010 0:00 /usr/sbin/sshd ganglia 3786 0.0 0.0 24704 988 ? Ss 2010 14:26 /usr/sbin/gmond root 3843 0.0 0.0 54144 300 ? Ss 2010 1:49 /usr/libexec/postfix/master postfix 3855 0.0 0.0 54860 1060 ? S 2010 0:22 qmgr -l -t fifo -u root 3877 0.0 0.0 74828 708 ? Ss 2010 1:15 crond root 3891 1.4 1.9 326960 77704 ? S<l 2010 896:59 mfschunkserver root 4122 0.0 0.0 18732 176 ? Ss 2010 0:10 /usr/sbin/atd root 4193 0.0 0.8 129180 35984 ? Ssl 2010 11:04 /usr/bin/ruby /usr/sbin/puppetd root 4223 0.0 0.0 18416 172 ? S 2010 0:10 /usr/sbin/smartd -q never root 4227 0.0 0.0 3792 8 tty1 Ss+ 2010 0:00 /sbin/mingetty tty1 root 4230 0.0 0.0 3792 8 tty2 Ss+ 2010 0:00 /sbin/mingetty tty2 root 4231 0.0 0.0 3792 8 tty3 Ss+ 2010 0:00 /sbin/mingetty tty3 root 4233 0.0 0.0 3792 8 tty4 Ss+ 2010 0:00 /sbin/mingetty tty4 root 4234 0.0 0.0 3792 8 tty5 Ss+ 2010 0:00 /sbin/mingetty tty5 root 4236 0.0 0.0 3792 8 tty6 Ss+ 2010 0:00 /sbin/mingetty tty6 root 5596 0.0 0.0 19368 20 ? Ss 2010 0:00 DarwinStreamingServer qtss 5597 0.8 0.9 166572 37408 ? Sl 2010 523:02 DarwinStreamingServer root 8714 0.0 0.0 0 0 ? S Jan31 0:33 [pdflush] root 9914 0.0 0.0 65612 968 pts/1 R+ 21:49 0:00 ps aux root 10765 0.0 0.0 76792 1080 ? Ss Jan24 0:58 SCREEN root 10766 0.0 0.0 66212 872 pts/3 Ss Jan24 0:00 /bin/bash root 11833 0.0 0.0 63852 1060 pts/3 S+ 17:17 0:00 /bin/sh ./launch.sh root 11834 437 42.9 4126884 1733348 pts/3 Sl+ 17:17 1190:50 /usr/bin/java -Xms128m -Xmx512m -XX:+UseConcMarkSweepGC -jar /JavaCore/JavaCore.jar root 13127 4.7 1.1 110564 46876 ? Ssl 17:18 12:55 /JavaCore/fetcher.bin root 19392 0.0 0.0 90108 3336 ? Rs 20:35 0:00 sshd: root@pts/1 root 19401 0.0 0.0 66216 1640 pts/1 Ss 20:35 0:00 -bash root 20567 0.0 0.0 90108 412 ? Ss Jan16 1:58 sshd: root@pts/0 root 20569 0.0 0.0 66084 912 pts/0 Ss Jan16 0:00 -bash root 21053 0.0 0.0 63856 28 ? S Jan30 0:00 /bin/sh /usr/bin/WowzaMediaServerd /usr/local/WowzaMediaServer/bin/setenv.sh /var/run/WowzaM root 21054 2.9 10.3 2252652 418468 ? Sl Jan30 314:25 java -Xmx1200M -server -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true - root 21915 0.0 0.0 0 0 ? S Feb01 0:00 [pdflush] root 29996 0.0 0.0 76524 1004 pts/0 S+ 14:41 0:00 screen -x Any idea what could this be, or where I should look for more diagnostic information? Thanks.

    Read the article

  • Server Security

    - by mahatmanich
    I want to run my own root server (directly accessible from the web without a hardware firewall) with debian lenny, apache2, php5, mysql, postfix MTA, sftp (based on ssh) and maybe dns server. What measures/software would you recomend, and why, to secure this server down and minimalize the attack vector? Webapplications aside ... This is what I have so far: iptables (for gen. packet filtering) fail2ban (brute force attack defense) ssh (chang default, port disable root access) modsecurity - is really clumsy and a pain (any alternative here?) ?Sudo why should I use it? what is the advantage to normal user handling thinking about greensql for mysql www.greensql.net is tripwire worth looking at? snort? What am I missing? What is hot and what is not? Best practices? I like "KISS" - Keep it simple secure, I know it would be nice! Thanks in advance ...

    Read the article

  • The blocking nature of aggregates

    - by Rob Farley
    I wrote a post recently about how query tuning isn’t just about how quickly the query runs – that if you have something (such as SSIS) that is consuming your data (and probably introducing a bottleneck), then it might be more important to have a query which focuses on getting the first bit of data out. You can read that post here.  In particular, we looked at two operators that could be used to ensure that a query returns only Distinct rows. and The Sort operator pulls in all the data, sorts it (discarding duplicates), and then pushes out the remaining rows. The Hash Match operator performs a Hashing function on each row as it comes in, and then looks to see if it’s created a Hash it’s seen before. If not, it pushes the row out. The Sort method is quicker, but has to wait until it’s gathered all the data before it can do the sort, and therefore blocks the data flow. But that was my last post. This one’s a bit different. This post is going to look at how Aggregate functions work, which ties nicely into this month’s T-SQL Tuesday. I’ve frequently explained about the fact that DISTINCT and GROUP BY are essentially the same function, although DISTINCT is the poorer cousin because you have less control over it, and you can’t apply aggregate functions. Just like the operators used for Distinct, there are different flavours of Aggregate operators – coming in blocking and non-blocking varieties. The example I like to use to explain this is a pile of playing cards. If I’m handed a pile of cards and asked to count how many cards there are in each suit, it’s going to help if the cards are already ordered. Suppose I’m playing a game of Bridge, I can easily glance at my hand and count how many there are in each suit, because I keep the pile of cards in order. Moving from left to right, I could tell you I have four Hearts in my hand, even before I’ve got to the end. By telling you that I have four Hearts as soon as I know, I demonstrate the principle of a non-blocking operation. This is known as a Stream Aggregate operation. It requires input which is sorted by whichever columns the grouping is on, and it will release a row as soon as the group changes – when I encounter a Spade, I know I don’t have any more Hearts in my hand. Alternatively, if the pile of cards are not sorted, I won’t know how many Hearts I have until I’ve looked through all the cards. In fact, to count them, I basically need to put them into little piles, and when I’ve finished making all those piles, I can count how many there are in each. Because I don’t know any of the final numbers until I’ve seen all the cards, this is blocking. This performs the aggregate function using a Hash Match. Observant readers will remember this from my Distinct example. You might remember that my earlier Hash Match operation – used for Distinct Flow – wasn’t blocking. But this one is. They’re essentially doing a similar operation, applying a Hash function to some data and seeing if the set of values have been seen before, but before, it needs more information than the mere existence of a new set of values, it needs to consider how many of them there are. A lot is dependent here on whether the data coming out of the source is sorted or not, and this is largely determined by the indexes that are being used. If you look in the Properties of an Index Scan, you’ll be able to see whether the order of the data is required by the plan. A property called Ordered will demonstrate this. In this particular example, the second plan is significantly faster, but is dependent on having ordered data. In fact, if I force a Stream Aggregate on unordered data (which I’m doing by telling it to use a different index), a Sort operation is needed, which makes my plan a lot slower. This is all very straight-forward stuff, and information that most people are fully aware of. I’m sure you’ve all read my good friend Paul White (@sql_kiwi)’s post on how the Query Optimizer chooses which type of aggregate function to apply. But let’s take a look at SQL Server Integration Services. SSIS gives us a Aggregate transformation for use in Data Flow Tasks, but it’s described as Blocking. The definitive article on Performance Tuning SSIS uses Sort and Aggregate as examples of Blocking Transformations. I’ve just shown you that Aggregate operations used by the Query Optimizer are not always blocking, but that the SSIS Aggregate component is an example of a blocking transformation. But is it always the case? After all, there are plenty of SSIS Performance Tuning talks out there that describe the value of sorted data in Data Flow Tasks, describing the IsSorted property that can be set through the Advanced Editor of your Source component. And so I set about testing the Aggregate transformation in SSIS, to prove for sure whether providing Sorted data would let the Aggregate transform behave like a Stream Aggregate. (Of course, I knew the answer already, but it helps to be able to demonstrate these things). A query that will produce a million rows in order was in order. Let me rephrase. I used a query which produced the numbers from 1 to 1000000, in a single field, ordered. The IsSorted flag was set on the source output, with the only column as SortKey 1. Performing an Aggregate function over this (counting the number of rows per distinct number) should produce an additional column with 1 in it. If this were being done in T-SQL, the ordered data would allow a Stream Aggregate to be used. In fact, if the Query Optimizer saw that the field had a Unique Index on it, it would be able to skip the Aggregate function completely, and just insert the value 1. This is a shortcut I wouldn’t be expecting from SSIS, but certainly the Stream behaviour would be nice. Unfortunately, it’s not the case. As you can see from the screenshots above, the data is pouring into the Aggregate function, and not being released until all million rows have been seen. It’s not doing a Stream Aggregate at all. This is expected behaviour. (I put that in bold, because I want you to realise this.) An SSIS transformation is a piece of code that runs. It’s a physical operation. When you write T-SQL and ask for an aggregation to be done, it’s a logical operation. The physical operation is either a Stream Aggregate or a Hash Match. In SSIS, you’re telling the system that you want a generic Aggregation, that will have to work with whatever data is passed in. I’m not saying that it wouldn’t be possible to make a sometimes-blocking aggregation component in SSIS. A Custom Component could be created which could detect whether the SortKeys columns of the input matched the Grouping columns of the Aggregation, and either call the blocking code or the non-blocking code as appropriate. One day I’ll make one of those, and publish it on my blog. I’ve done it before with a Script Component, but as Script components are single-use, I was able to handle the data knowing everything about my data flow already. As per my previous post – there are a lot of aspects in which tuning SSIS and tuning execution plans use similar concepts. In both situations, it really helps to have a feel for what’s going on behind the scenes. Considering whether an operation is blocking or not is extremely relevant to performance, and that it’s not always obvious from the surface. In a future post, I’ll show the impact of blocking v non-blocking and synchronous v asynchronous components in SSIS, using some of LobsterPot’s Script Components and Custom Components as examples. When I get that sorted, I’ll make a Stream Aggregate component available for download.

    Read the article

  • The blocking nature of aggregates

    - by Rob Farley
    I wrote a post recently about how query tuning isn’t just about how quickly the query runs – that if you have something (such as SSIS) that is consuming your data (and probably introducing a bottleneck), then it might be more important to have a query which focuses on getting the first bit of data out. You can read that post here.  In particular, we looked at two operators that could be used to ensure that a query returns only Distinct rows. and The Sort operator pulls in all the data, sorts it (discarding duplicates), and then pushes out the remaining rows. The Hash Match operator performs a Hashing function on each row as it comes in, and then looks to see if it’s created a Hash it’s seen before. If not, it pushes the row out. The Sort method is quicker, but has to wait until it’s gathered all the data before it can do the sort, and therefore blocks the data flow. But that was my last post. This one’s a bit different. This post is going to look at how Aggregate functions work, which ties nicely into this month’s T-SQL Tuesday. I’ve frequently explained about the fact that DISTINCT and GROUP BY are essentially the same function, although DISTINCT is the poorer cousin because you have less control over it, and you can’t apply aggregate functions. Just like the operators used for Distinct, there are different flavours of Aggregate operators – coming in blocking and non-blocking varieties. The example I like to use to explain this is a pile of playing cards. If I’m handed a pile of cards and asked to count how many cards there are in each suit, it’s going to help if the cards are already ordered. Suppose I’m playing a game of Bridge, I can easily glance at my hand and count how many there are in each suit, because I keep the pile of cards in order. Moving from left to right, I could tell you I have four Hearts in my hand, even before I’ve got to the end. By telling you that I have four Hearts as soon as I know, I demonstrate the principle of a non-blocking operation. This is known as a Stream Aggregate operation. It requires input which is sorted by whichever columns the grouping is on, and it will release a row as soon as the group changes – when I encounter a Spade, I know I don’t have any more Hearts in my hand. Alternatively, if the pile of cards are not sorted, I won’t know how many Hearts I have until I’ve looked through all the cards. In fact, to count them, I basically need to put them into little piles, and when I’ve finished making all those piles, I can count how many there are in each. Because I don’t know any of the final numbers until I’ve seen all the cards, this is blocking. This performs the aggregate function using a Hash Match. Observant readers will remember this from my Distinct example. You might remember that my earlier Hash Match operation – used for Distinct Flow – wasn’t blocking. But this one is. They’re essentially doing a similar operation, applying a Hash function to some data and seeing if the set of values have been seen before, but before, it needs more information than the mere existence of a new set of values, it needs to consider how many of them there are. A lot is dependent here on whether the data coming out of the source is sorted or not, and this is largely determined by the indexes that are being used. If you look in the Properties of an Index Scan, you’ll be able to see whether the order of the data is required by the plan. A property called Ordered will demonstrate this. In this particular example, the second plan is significantly faster, but is dependent on having ordered data. In fact, if I force a Stream Aggregate on unordered data (which I’m doing by telling it to use a different index), a Sort operation is needed, which makes my plan a lot slower. This is all very straight-forward stuff, and information that most people are fully aware of. I’m sure you’ve all read my good friend Paul White (@sql_kiwi)’s post on how the Query Optimizer chooses which type of aggregate function to apply. But let’s take a look at SQL Server Integration Services. SSIS gives us a Aggregate transformation for use in Data Flow Tasks, but it’s described as Blocking. The definitive article on Performance Tuning SSIS uses Sort and Aggregate as examples of Blocking Transformations. I’ve just shown you that Aggregate operations used by the Query Optimizer are not always blocking, but that the SSIS Aggregate component is an example of a blocking transformation. But is it always the case? After all, there are plenty of SSIS Performance Tuning talks out there that describe the value of sorted data in Data Flow Tasks, describing the IsSorted property that can be set through the Advanced Editor of your Source component. And so I set about testing the Aggregate transformation in SSIS, to prove for sure whether providing Sorted data would let the Aggregate transform behave like a Stream Aggregate. (Of course, I knew the answer already, but it helps to be able to demonstrate these things). A query that will produce a million rows in order was in order. Let me rephrase. I used a query which produced the numbers from 1 to 1000000, in a single field, ordered. The IsSorted flag was set on the source output, with the only column as SortKey 1. Performing an Aggregate function over this (counting the number of rows per distinct number) should produce an additional column with 1 in it. If this were being done in T-SQL, the ordered data would allow a Stream Aggregate to be used. In fact, if the Query Optimizer saw that the field had a Unique Index on it, it would be able to skip the Aggregate function completely, and just insert the value 1. This is a shortcut I wouldn’t be expecting from SSIS, but certainly the Stream behaviour would be nice. Unfortunately, it’s not the case. As you can see from the screenshots above, the data is pouring into the Aggregate function, and not being released until all million rows have been seen. It’s not doing a Stream Aggregate at all. This is expected behaviour. (I put that in bold, because I want you to realise this.) An SSIS transformation is a piece of code that runs. It’s a physical operation. When you write T-SQL and ask for an aggregation to be done, it’s a logical operation. The physical operation is either a Stream Aggregate or a Hash Match. In SSIS, you’re telling the system that you want a generic Aggregation, that will have to work with whatever data is passed in. I’m not saying that it wouldn’t be possible to make a sometimes-blocking aggregation component in SSIS. A Custom Component could be created which could detect whether the SortKeys columns of the input matched the Grouping columns of the Aggregation, and either call the blocking code or the non-blocking code as appropriate. One day I’ll make one of those, and publish it on my blog. I’ve done it before with a Script Component, but as Script components are single-use, I was able to handle the data knowing everything about my data flow already. As per my previous post – there are a lot of aspects in which tuning SSIS and tuning execution plans use similar concepts. In both situations, it really helps to have a feel for what’s going on behind the scenes. Considering whether an operation is blocking or not is extremely relevant to performance, and that it’s not always obvious from the surface. In a future post, I’ll show the impact of blocking v non-blocking and synchronous v asynchronous components in SSIS, using some of LobsterPot’s Script Components and Custom Components as examples. When I get that sorted, I’ll make a Stream Aggregate component available for download.

    Read the article

  • Do I need to store a generic rotation point/radius for rotating around a point other than the origin for object transforms?

    - by Casey
    I'm having trouble implementing a non-origin point rotation. I have a class Transform that stores each component separately in three 3D vectors for position, scale, and rotation. This is fine for local rotations based on the center of the object. The issue is how do I determine/concatenate non-origin rotations in addition to origin rotations. Normally this would be achieved as a Transform-Rotate-Transform for the center rotation followed by a Transform-Rotate-Transform for the non-origin point. The problem is because I am storing the individual components, the final Transform matrix is not calculated until needed by using the individual components to fill an appropriate Matrix. (See GetLocalTransform()) Do I need to store an additional rotation (and radius) for world rotations as well or is there a method of implementation that works while only using the single rotation value? Transform.h #ifndef A2DE_CTRANSFORM_H #define A2DE_CTRANSFORM_H #include "../a2de_vals.h" #include "CMatrix4x4.h" #include "CVector3D.h" #include <vector> A2DE_BEGIN class Transform { public: Transform(); Transform(Transform* parent); Transform(const Transform& other); Transform& operator=(const Transform& rhs); virtual ~Transform(); void SetParent(Transform* parent); void AddChild(Transform* child); void RemoveChild(Transform* child); Transform* FirstChild(); Transform* LastChild(); Transform* NextChild(); Transform* PreviousChild(); Transform* GetChild(std::size_t index); std::size_t GetChildCount() const; std::size_t GetChildCount(); void SetPosition(const a2de::Vector3D& position); const a2de::Vector3D& GetPosition() const; a2de::Vector3D& GetPosition(); void SetRotation(const a2de::Vector3D& rotation); const a2de::Vector3D& GetRotation() const; a2de::Vector3D& GetRotation(); void SetScale(const a2de::Vector3D& scale); const a2de::Vector3D& GetScale() const; a2de::Vector3D& GetScale(); a2de::Matrix4x4 GetLocalTransform() const; a2de::Matrix4x4 GetLocalTransform(); protected: private: a2de::Vector3D _position; a2de::Vector3D _scale; a2de::Vector3D _rotation; std::size_t _curChildIndex; Transform* _parent; std::vector<Transform*> _children; }; A2DE_END #endif Transform.cpp #include "CTransform.h" #include "CVector2D.h" #include "CVector4D.h" A2DE_BEGIN Transform::Transform() : _position(), _scale(1.0, 1.0), _rotation(), _curChildIndex(0), _parent(nullptr), _children() { /* DO NOTHING */ } Transform::Transform(Transform* parent) : _position(), _scale(1.0, 1.0), _rotation(), _curChildIndex(0), _parent(parent), _children() { /* DO NOTHING */ } Transform::Transform(const Transform& other) : _position(other._position), _scale(other._scale), _rotation(other._rotation), _curChildIndex(0), _parent(other._parent), _children(other._children) { /* DO NOTHING */ } Transform& Transform::operator=(const Transform& rhs) { if(this == &rhs) return *this; this->_position = rhs._position; this->_scale = rhs._scale; this->_rotation = rhs._rotation; this->_curChildIndex = 0; this->_parent = rhs._parent; this->_children = rhs._children; return *this; } Transform::~Transform() { _children.clear(); _parent = nullptr; } void Transform::SetParent(Transform* parent) { _parent = parent; } void Transform::AddChild(Transform* child) { if(child == nullptr) return; _children.push_back(child); } void Transform::RemoveChild(Transform* child) { if(_children.empty()) return; _children.erase(std::remove(_children.begin(), _children.end(), child), _children.end()); } Transform* Transform::FirstChild() { if(_children.empty()) return nullptr; return *(_children.begin()); } Transform* Transform::LastChild() { if(_children.empty()) return nullptr; return *(_children.end()); } Transform* Transform::NextChild() { if(_children.empty()) return nullptr; std::size_t s(_children.size()); if(_curChildIndex >= s) { _curChildIndex = s; return nullptr; } return _children[_curChildIndex++]; } Transform* Transform::PreviousChild() { if(_children.empty()) return nullptr; if(_curChildIndex == 0) { return nullptr; } return _children[_curChildIndex--]; } Transform* Transform::GetChild(std::size_t index) { if(_children.empty()) return nullptr; if(index > _children.size()) return nullptr; return _children[index]; } std::size_t Transform::GetChildCount() const { if(_children.empty()) return 0; return _children.size(); } std::size_t Transform::GetChildCount() { return static_cast<const Transform&>(*this).GetChildCount(); } void Transform::SetPosition(const a2de::Vector3D& position) { _position = position; } const a2de::Vector3D& Transform::GetPosition() const { return _position; } a2de::Vector3D& Transform::GetPosition() { return const_cast<a2de::Vector3D&>(static_cast<const Transform&>(*this).GetPosition()); } void Transform::SetRotation(const a2de::Vector3D& rotation) { _rotation = rotation; } const a2de::Vector3D& Transform::GetRotation() const { return _rotation; } a2de::Vector3D& Transform::GetRotation() { return const_cast<a2de::Vector3D&>(static_cast<const Transform&>(*this).GetRotation()); } void Transform::SetScale(const a2de::Vector3D& scale) { _scale = scale; } const a2de::Vector3D& Transform::GetScale() const { return _scale; } a2de::Vector3D& Transform::GetScale() { return const_cast<a2de::Vector3D&>(static_cast<const Transform&>(*this).GetScale()); } a2de::Matrix4x4 Transform::GetLocalTransform() const { Matrix4x4 p((_parent ? _parent->GetLocalTransform() : a2de::Matrix4x4::GetIdentity())); Matrix4x4 t(a2de::Matrix4x4::GetTranslationMatrix(_position)); Matrix4x4 r(a2de::Matrix4x4::GetRotationMatrix(_rotation)); Matrix4x4 s(a2de::Matrix4x4::GetScaleMatrix(_scale)); return (p * t * r * s); } a2de::Matrix4x4 Transform::GetLocalTransform() { return static_cast<const Transform&>(*this).GetLocalTransform(); } A2DE_END

    Read the article

  • PostSharp, Obfuscation, and IL

    - by Simon Cooper
    Aspect-oriented programming (AOP) is a relatively new programming paradigm. Originating at Xerox PARC in 1994, the paradigm was first made available for general-purpose development as an extension to Java in 2001. From there, it has quickly been adapted for use in all the common languages used today. In the .NET world, one of the primary AOP toolkits is PostSharp. Attributes and AOP Normally, attributes in .NET are entirely a metadata construct. Apart from a few special attributes in the .NET framework, they have no effect whatsoever on how a class or method executes within the CLR. Only by using reflection at runtime can you access any attributes declared on a type or type member. PostSharp changes this. By declaring a custom attribute that derives from PostSharp.Aspects.Aspect, applying it to types and type members, and running the resulting assembly through the PostSharp postprocessor, you can essentially declare 'clever' attributes that change the behaviour of whatever the aspect has been applied to at runtime. A simple example of this is logging. By declaring a TraceAttribute that derives from OnMethodBoundaryAspect, you can automatically log when a method has been executed: public class TraceAttribute : PostSharp.Aspects.OnMethodBoundaryAspect { public override void OnEntry(MethodExecutionArgs args) { MethodBase method = args.Method; System.Diagnostics.Trace.WriteLine( String.Format( "Entering {0}.{1}.", method.DeclaringType.FullName, method.Name)); } public override void OnExit(MethodExecutionArgs args) { MethodBase method = args.Method; System.Diagnostics.Trace.WriteLine( String.Format( "Leaving {0}.{1}.", method.DeclaringType.FullName, method.Name)); } } [Trace] public void MethodToLog() { ... } Now, whenever MethodToLog is executed, the aspect will automatically log entry and exit, without having to add the logging code to MethodToLog itself. PostSharp Performance Now this does introduce a performance overhead - as you can see, the aspect allows access to the MethodBase of the method the aspect has been applied to. If you were limited to C#, you would be forced to retrieve each MethodBase instance using Type.GetMethod(), matching on the method name and signature. This is slow. Fortunately, PostSharp is not limited to C#. It can use any instruction available in IL. And in IL, you can do some very neat things. Ldtoken C# allows you to get the Type object corresponding to a specific type name using the typeof operator: Type t = typeof(Random); The C# compiler compiles this operator to the following IL: ldtoken [mscorlib]System.Random call class [mscorlib]System.Type [mscorlib]System.Type::GetTypeFromHandle( valuetype [mscorlib]System.RuntimeTypeHandle) The ldtoken instruction obtains a special handle to a type called a RuntimeTypeHandle, and from that, the Type object can be obtained using GetTypeFromHandle. These are both relatively fast operations - no string lookup is required, only direct assembly and CLR constructs are used. However, a little-known feature is that ldtoken is not just limited to types; it can also get information on methods and fields, encapsulated in a RuntimeMethodHandle or RuntimeFieldHandle: // get a MethodBase for String.EndsWith(string) ldtoken method instance bool [mscorlib]System.String::EndsWith(string) call class [mscorlib]System.Reflection.MethodBase [mscorlib]System.Reflection.MethodBase::GetMethodFromHandle( valuetype [mscorlib]System.RuntimeMethodHandle) // get a FieldInfo for the String.Empty field ldtoken field string [mscorlib]System.String::Empty call class [mscorlib]System.Reflection.FieldInfo [mscorlib]System.Reflection.FieldInfo::GetFieldFromHandle( valuetype [mscorlib]System.RuntimeFieldHandle) These usages of ldtoken aren't usable from C# or VB, and aren't likely to be added anytime soon (Eric Lippert's done a blog post on the possibility of adding infoof, methodof or fieldof operators to C#). However, PostSharp deals directly with IL, and so can use ldtoken to get MethodBase objects quickly and cheaply, without having to resort to string lookups. The kicker However, there are problems. Because ldtoken for methods or fields isn't accessible from C# or VB, it hasn't been as well-tested as ldtoken for types. This has resulted in various obscure bugs in most versions of the CLR when dealing with ldtoken and methods, and specifically, generic methods and methods of generic types. This means that PostSharp was behaving incorrectly, or just plain crashing, when aspects were applied to methods that were generic in some way. So, PostSharp has to work around this. Without using the metadata tokens directly, the only way to get the MethodBase of generic methods is to use reflection: Type.GetMethod(), passing in the method name as a string along with information on the signature. Now, this works fine. It's slower than using ldtoken directly, but it works, and this only has to be done for generic methods. Unfortunately, this poses problems when the assembly is obfuscated. PostSharp and Obfuscation When using ldtoken, obfuscators don't affect how PostSharp operates. Because the ldtoken instruction directly references the type, method or field within the assembly, it is unaffected if the name of the object is changed by an obfuscator. However, the indirect loading used for generic methods was breaking, because that uses the name of the method when the assembly is put through the PostSharp postprocessor to lookup the MethodBase at runtime. If the name then changes, PostSharp can't find it anymore, and the assembly breaks. So, PostSharp needs to know about any changes an obfuscator does to an assembly. The way PostSharp does this is by adding another layer of indirection. When PostSharp obfuscation support is enabled, it includes an extra 'name table' resource in the assembly, consisting of a series of method & type names. When PostSharp needs to lookup a method using reflection, instead of encoding the method name directly, it looks up the method name at a fixed offset inside that name table: MethodBase genericMethod = typeof(ContainingClass).GetMethod(GetNameAtIndex(22)); PostSharp.NameTable resource: ... 20: get_Prop1 21: set_Prop1 22: DoFoo 23: GetWibble When the assembly is later processed by an obfuscator, the obfuscator can replace all the method and type names within the name table with their new name. That way, the reflection lookups performed by PostSharp will now use the new names, and everything will work as expected: MethodBase genericMethod = typeof(#kGy).GetMethod(GetNameAtIndex(22)); PostSharp.NameTable resource: ... 20: #kkA 21: #zAb 22: #EF5a 23: #2tg As you can see, this requires direct support by an obfuscator in order to perform these rewrites. Dotfuscator supports it, and now, starting with SmartAssembly 6.6.4, SmartAssembly does too. So, a relatively simple solution to a tricky problem, with some CLR bugs thrown in for good measure. You don't see those every day!

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 92 93 94 95 96 97 98 99 100 101 102 103  | Next Page >