parallel foreach - Page 36

Parallelism: Who Cares?

Parallel computing - Programming - Languages - Hardware - C++

i have a script that get all the info i need about my SharePoint farm : [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint") > $null $farm = [Microsoft.SharePoint.Administration.SPFarm]::Local $websvcs = $farm.Services | where -FilterScript {$_.GetType() -eq [Microsoft.SharePoint.Administration.SPWebService]} $webapps = @() foreach ($websvc in $websvcs) { write-output "Web Applications" write-output "" foreach ($webapp in $websvc.WebApplications) { write-output "Webapp Name -->"$webapp.Name write-output "" write-output "Site Collections" write-output "" foreach ($site in $webapp.Sites) { write-output "Site URL --> -->" $site.URL write-output "" write-output "Websites" write-output "" foreach ($web in $site.AllWebs) { write-output "Web URL --> --> -->" $web.URL write-output "" write-output "Lists" write-output "" foreach ($list in $web.Lists) { write-output "List Title --> --> --> -->" $list.Title write-output "" } foreach ($group in $web.Groups) { write-output "Group Name --> --> --> -->" $group.Name write-output "" foreach ($user in $group.Users) { write-output "User Name --> --> --> -->" $user.Name write-output "" } } } } } } i want to make the output to an XML file and then connect the xml file to HTML and make a site of it for manager use how can i do it ? thanks for the help !

Read the article

Does "for" in .Net Framework 4.0 execute loops in parallel? Or why is the total not the sum of the p

- by Shiraz Bhaiji

I am writing code to performance test a web site. I have the following code: string url = "http://xxxxxx"; System.Diagnostics.Stopwatch stopwatch = new System.Diagnostics.Stopwatch(); System.Diagnostics.Stopwatch totalTime = new System.Diagnostics.Stopwatch(); totalTime.Start(); for (int i = 0; i < 10; i++) { stopwatch.Start(); WebRequest request = HttpWebRequest.Create(url); WebResponse webResponse = request.GetResponse(); webResponse.Close(); stopwatch.Stop(); textBox1.Text += "Time Taken " + i.ToString() + " = " + stopwatch.Elapsed.Milliseconds.ToString() + Environment.NewLine; stopwatch.Reset(); } totalTime.Stop(); textBox1.Text += "Total Time Taken = " + totalTime.Elapsed.Milliseconds.ToString() + Environment.NewLine; Which is giving the following result: Time Taken 0 = 88 Time Taken 1 = 161 Time Taken 2 = 218 Time Taken 3 = 417 Time Taken 4 = 236 Time Taken 5 = 217 Time Taken 6 = 217 Time Taken 7 = 218 Time Taken 8 = 409 Time Taken 9 = 48 Total Time Taken = 257 I had expected the total time to be the sum of the individual times. Can anybody see why it is not?

Read the article

How to execute a program on PostBuild event in parallel?

- by John

I managed to set the compiler to execute another program when the project is built/ran with the following directive in project options: call program.exe param1 param2 The problem is that the compiler executes "program.exe" and waits for it to terminate and THEN the project executable is ran. What I ask: How to set the compiler to run both executables in paralel without waiting for the one in PostBuild event to terminate? Thanks in advance

Read the article

SSIS - Parallel Execution of Tasks - How efficient is it?

- by Randy Minder

I am building an SSIS package that will contain dozens of Sequence tasks. Each Sequence task will contain three tasks. One to truncate a destination table and remove indexes on the table, another to import data from a source table, and a third to add back indexes to the destination table. My question is this. I currently have nine of these Sequences tasks built, and none are dependent on any of the others. When I execute the package, SSIS seems to do a pretty good job of determining which tasks in which Sequence to execute, which, by the way, appears to be quite random. As I continue adding more Sequences, should I attempt to be smarter about how SSIS should execute these Sequences, or is SSIS smart enough to do it itself? Thanks.

Read the article

LINQ - How to change value in select or foreach loop ?

- by Eugene

IQueryable items = from rr in _dt.AllItems where rr.ID == ItemID select new{rr.Item, rr.SecondItem}; SecondItem is returning false or true. Is it possible to replace with different string values (for example with On and Off) and add to the items (IQueryable)?

Read the article

Resharper Unit Test Runner executes unit tests sequentially or in parallel?

- by Kostas

I would like to know if Resharper Unit Test runner runs unit tests sequentially or in parallell (each on its own thread). I am using MBUnit test framework. Thanks.

Read the article

In Perl, how can I wait for threads to end in parallel?

- by Pmarcoen

I have a Perl script that launches 2 threads,one for each processor. I need it to wait for a thread to end, if one thread ends a new one is spawned. It seems that the join method blocks the rest of the program, therefore the second thread can't end until everything the first thread does is done which sort of defeats its purpose. I tried the is_joinable method but that doesn't seem to do it either. Here is some of my code : use threads; use threads::shared; @file_list = @ARGV; #Our file list $nofiles = $#file_list + 1; #Real number of files $currfile = 1; #Current number of file to process my %MSG : shared; #shared hash $thr0 = threads->new(\&process, shift(@file_list)); $currfile++; $thr1 = threads->new(\&process, shift(@file_list)); $currfile++; while(1){ if ($thr0->is_joinable()) { $thr0->join; #check if there are files left to process if($currfile <= $nofiles){ $thr0 = threads->new(\&process, shift(@file_list)); $currfile++; } } if ($thr1->is_joinable()) { $thr1->join; #check if there are files left to process if($currfile <= $nofiles){ $thr1 = threads->new(\&process, shift(@file_list)); $currfile++; } } } sub process{ print "Opening $currfile of $nofiles\n"; #do some stuff if(some condition){ lock(%MSG); #write stuff to hash } print "Closing $currfile of $nofiles\n"; } The output of this is : Opening 1 of 4 Opening 2 of 4 Closing 1 of 4 Opening 3 of 4 Closing 3 of 4 Opening 4 of 4 Closing 2 of 4 Closing 4 of 4

Read the article

Parallel Dev: Should developers work within the same branch?

- by Zombies

Should multiple developers work within the same branch, and update - modify - commit ? Or should each developer have his/her own each branch exclusively? And how would sharing branches impact an environment where you are doing routine maintenance as opposed to unmaintained code streams? Also, how would this work if you deploy each developers work as soon as it is done and passes testing (rapidly, as opposed to putting all of their work into a single release).

Read the article

SSIS: Way to handle hot folder items in parallel?

- by Dr. Zim

We have eight Xeon (i7) cores and 16 gig of RAM on our SSIS box. We have about 200 image files we want to convert using a command line utility every day. Currently the process is using Adobe Photoshop and droplets (very manual, taking upwards of two hours a day) Using SSIS hot folders, is there a way to execute up to eight conversions at once? Is there any way to tell a process completed or execute code upon it's completion?

Read the article

Should I be using the for-loop construct or the forEach method for iteration?

- by Dutch

Is one way more efficient? Are there any limitations in one method versus the other? Thanks for any input you guys can give me ;-)

Read the article

How (and if) to write a single-consumer queue using the task parallel library?

- by Eric

I've heard a bunch of podcasts recently about the TPL in .NET 4.0. Most of them describe background activities like downloading images or doing a computation, using tasks so that the work doesn't interfere with a GUI thread. Most of the code I work on has more of a multiple-producer / single-consumer flavor, where work items from multiple sources must be queued and then processed in order. One example would be logging, where log lines from multiple threads are sequentialized into a single queue for eventual writing to a file or database. All the records from any single source must remain in order, and records from the same moment in time should be "close" to each other in the eventual output. So multiple threads or tasks or whatever are all invoking a queuer: lock( _queue ) // or use a lock-free queue! { _queue.enqueue( some_work ); _queueSemaphore.Release(); } And a dedicated worker thread processes the queue: while( _queueSemaphore.WaitOne() ) { lock( _queue ) { some_work = _queue.dequeue(); } deal_with( some_work ); } It's always seemed reasonable to dedicate a worker thread for the consumer side of these tasks. Should I write future programs using some construct from the TPL instead? Which one? Why?

Read the article

in Javascript, should I be using the for-loop construct or the forEach method for iteration?

- by Dutch

Is one way more efficient? Are there any limitations in one method versus the other? Thanks for any input you guys can give me ;-)

Read the article

Are parallel calls to send/recv on the same socket valid?

- by Jay

Can we call send from one thread and recv from another on the same socket? Can we call multiple sends parallely from different threads on the same socket? I know that a good design should avoid this, but I am not clear how these system APIs will behave. I am unable to find a good documentation also for the same. Any pointers in the direction will be helpful.

Read the article

Parallel MySQL queries for HTML table - WHILE(x or y)?

- by Beti Chode

I'm trying to create a table using PHP. What I need is a table with two columns. So I have an SQL table with 4 fields - primary key id, language, word and definition. The language for each is either Arabic or Russian. I want a table that does the following: | defintion | |____________________| | | | rus1 | arab1 | | rus2 | arab2 | | rus3 | arab3 | | rus4 | | So it divides the list by English word, creates a for each English word, then lists Russian equivalents in the left column and Arabic in the right. However there are often not the same number for both. What I am doing right now is running a WHILE loop in a WHILE loop. The outer loop is running fine but I think I am doing the inner loop wrong. Here is the bulk of the code: $definitions=mysql_query("SELECT DISTINCT definition FROM words") WHILE($row=mysql_fetch_array($definitions) { ECHO '<tr><th colspan="2">' . $row['definition'] . '</th></tr>'; $russian="SELECT * FROM words WHERE language='Russian' AND definition='".$row['definition']."'"; $arabic="SELECT * FROM words WHERE language='Arabic' AND definition='".$row['definition']."'"; WHILE($rus=mysql_fetch_array($russian) or $arb=mysql_fetch_array($arabic)) { ECHO '<tr><td>'.$rus['word'].'</td><td>'.$arb['word'].'</td></tr>'; } } Sadly I am getting soemthing like this: | defintion | |____________________| | | | rus1 | | | rus2 | | | rus3 | | | rus4 | | | | arab1 | | | arab2 | | | arab3 | Not sure what other way I can do this? I tried changing the or to || thinking the different precedence would cause another outcome, but then I get ONLY the Russian column. I'm out of ideas, you guys are my only hope!

Read the article

Doing a generic <sql:query> in Grails

- by melling

This is a generic way to select data from a table and show the results in an HTML table using JSP taglibs. What is the generic way to do this in Grails? That is, take a few lines of SQL and generate an HTML table from scratch in Grails, including the column names as headers. <sql:query var="results" dataSource="${dsource}" select * from foo </sql:query (# of rows: ${results.rowCount}) <table border="1" <!-- column headers -- <tr bgcolor=cyan <c:forEach var="columnName" items="${results.columnNames}" <th<c:out value="${columnName}"/</th </c:forEach </tr <!-- column data -- <c:forEach var="row" items="${results.rowsByIndex}" <tr <c:forEach var="column" items="${row}" <td<c:out value="${column}"/</td </c:forEach </tr </c:forEach </table The solution to this was answered in another StackOverFlow question. http://stackoverflow.com/questions/425294/sql-database-views-in-grails IF SOMEONE WRITES A GOOD ANSWER, I'LL ACCEPT IT. I would like a 100% acceptance on all of my questions.

Read the article

Why is casting and comparing in PHP faster than is_*?

- by tstenner

While optimizing a function in PHP, I changed if(is_array($obj)) foreach($obj as $key=$value { [snip] } else if(is_object($obj)) foreach($obj as $key=$value { [snip] } to if($obj == (array) $obj) foreach($obj as $key=$value { [snip] } else if($obj == (obj) $obj) foreach($obj as $key=$value { [snip] } After learning about ===, I changed that to if($obj === (array) $obj) foreach($obj as $key=$value { [snip] } else if($obj === (obj) $obj) foreach($obj as $key=$value { [snip] } Changing each test from is_* to casting resulted in a major speedup (30%). I understand that === is faster than == as no coercion has to be done, but why is casting the variable so much faster than calling any of the is_*-functions?

Read the article

Much Ado About Nothing: Stub Objects

- by user9154181

The Solaris 11 link-editor (ld) contains support for a new type of object that we call a stub object. A stub object is a shared object, built entirely from mapfiles, that supplies the same linking interface as the real object, while containing no code or data. Stub objects cannot be executed — the runtime linker will kill any process that attempts to load one. However, you can link to a stub object as a dependency, allowing the stub to act as a proxy for the real version of the object. You may well wonder if there is a point to producing an object that contains nothing but linking interface. As it turns out, stub objects are very useful for building large bodies of code such as Solaris. In the last year, we've had considerable success in applying them to one of our oldest and thorniest build problems. In this discussion, I will describe how we came to invent these objects, and how we apply them to building Solaris. This posting explains where the idea for stub objects came from, and details our long and twisty journey from hallway idea to standard link-editor feature. I expect that these details are mainly of interest to those who work on Solaris and its makefiles, those who have done so in the past, and those who work with other similar bodies of code. A subsequent posting will omit the history and background details, and instead discuss how to build and use stub objects. If you are mainly interested in what stub objects are, and don't care about the underlying software war stories, I encourage you to skip ahead. The Long Road To Stubs This all started for me with an email discussion in May of 2008, regarding a change request that was filed in 2002, entitled: 4631488 lib/Makefile is too patient: .WAITs should be reduced This CR encapsulates a number of cronic issues with Solaris builds: We build Solaris with a parallel make (dmake) that tries to build as much of the code base in parallel as possible. There is a lot of code to build, and we've long made use of parallelized builds to get the job done quicker. This is even more important in today's world of massively multicore hardware. Solaris contains a large number of executables and shared objects. Executables depend on shared objects, and shared objects can depend on each other. Before you can build an object, you need to ensure that the objects it needs have been built. This implies a need for serialization, which is in direct opposition to the desire to build everying in parallel. To accurately build objects in the right order requires an accurate set of make rules defining the things that depend on each other. This sounds simple, but the reality is quite complex. In practice, having programmers explicitly specify these dependencies is a losing strategy: It's really hard to get right. It's really easy to get it wrong and never know it because things build anyway. Even if you get it right, it won't stay that way, because dependencies between objects can change over time, and make cannot help you detect such drifing. You won't know that you got it wrong until the builds break. That can be a long time after the change that triggered the breakage happened, making it hard to connect the cause and the effect. Usually this happens just before a release, when the pressure is on, its hard to think calmly, and there is no time for deep fixes. As a poor compromise, the libraries in core Solaris were built using a set of grossly incomplete hand written rules, supplemented with a number of dmake .WAIT directives used to group the libraries into sets of non-interacting groups that can be built in parallel because we think they don't depend on each other. From time to time, someone will suggest that we could analyze the built objects themselves to determine their dependencies and then generate make rules based on those relationships. This is possible, but but there are complications that limit the usefulness of that approach: To analyze an object, you have to build it first. This is a classic chicken and egg scenario. You could analyze the results of a previous build, but then you're not necessarily going to get accurate rules for the current code. It should be possible to build the code without having a built workspace available. The analysis will take time, and remember that we're constantly trying to make builds faster, not slower. By definition, such an approach will always be approximate, and therefore only incremantally more accurate than the hand written rules described above. The hand written rules are fast and cheap, while this idea is slow and complex, so we stayed with the hand written approach. Solaris was built that way, essentially forever, because these are genuinely difficult problems that had no easy answer. The makefiles were full of build races in which the right outcomes happened reliably for years until a new machine or a change in build server workload upset the accidental balance of things. After figuring out what had happened, you'd mutter "How did that ever work?", add another incomplete and soon to be inaccurate make dependency rule to the system, and move on. This was not a satisfying solution, as we tend to be perfectionists in the Solaris group, but we didn't have a better answer. It worked well enough, approximately. And so it went for years. We needed a different approach — a new idea to cut the Gordian Knot. In that discussion from May 2008, my fellow linker-alien Rod Evans had the initial spark that lead us to a game changing series of realizations: The link-editor is used to link objects together, but it only uses the ELF metadata in the object, consisting of symbol tables, ELF versioning sections, and similar data. Notably, it does not look at, or understand, the machine code that makes an object useful at runtime. If you had an object that only contained the ELF metadata for a dependency, but not the code or data, the link-editor would find it equally useful for linking, and would never know the difference. Call it a stub object. In the core Solaris OS, we require all objects to be built with a link-editor mapfile that describes all of its publically available functions and data. Could we build a stub object using the mapfile for the real object? It ought to be very fast to build stub objects, as there are no input objects to process. Unlike the real object, stub objects would not actually require any dependencies, and so, all of the stubs for the entire system could be built in parallel. When building the real objects, one could link against the stub objects instead of the real dependencies. This means that all the real objects can be built built in parallel too, without any serialization. We could replace a system that requires perfect makefile rules with a system that requires no ordering rules whatsoever. The results would be considerably more robust. We immediately realized that this idea had potential, but also that there were many details to sort out, lots of work to do, and that perhaps it wouldn't really pan out. As is often the case, it would be necessary to do the work and see how it turned out. Following that conversation, I set about trying to build a stub object. We determined that a faithful stub has to do the following: Present the same set of global symbols, with the same ELF versioning, as the real object. Functions are simple — it suffices to have a symbol of the right type, possibly, but not necessarily, referencing a null function in its text segment. Copy relocations make data more complicated to stub. The possibility of a copy relocation means that when you create a stub, the data symbols must have the actual size of the real data. Any error in this will go uncaught at link time, and will cause tragic failures at runtime that are very hard to diagnose. For reasons too obscure to go into here, involving tentative symbols, it is also important that the data reside in bss, or not, matching its placement in the real object. If the real object has more than one symbol pointing at the same data item, we call these aliased symbols. All data symbols in the stub object must exhibit the same aliasing as the real object. We imagined the stub library feature working as follows: A command line option to ld tells it to produce a stub rather than a real object. In this mode, only mapfiles are examined, and any object or shared libraries on the command line are are ignored. The extra information needed (function or data, size, and bss details) would be added to the mapfile. When building the real object instead of the stub, the extra information for building stubs would be validated against the resulting object to ensure that they match. In exploring these ideas, I immediately run headfirst into the reality of the original mapfile syntax, a subject that I would later write about as The Problem(s) With Solaris SVR4 Link-Editor Mapfiles. The idea of extending that poor language was a non-starter. Until a better mapfile syntax became available, which seemed unlikely in 2008, the solution could not involve extentions to the mapfile syntax. Instead, we cooked up the idea (hack) of augmenting mapfiles with stylized comments that would carry the necessary information. A typical definition might look like: # DATA(i386) __iob 0x3c0 # DATA(amd64,sparcv9) __iob 0xa00 # DATA(sparc) __iob 0x140 iob; A further problem then became clear: If we can't extend the mapfile syntax, then there's no good way to extend ld with an option to produce stub objects, and to validate them against the real objects. The idea of having ld read comments in a mapfile and parse them for content is an unacceptable hack. The entire point of comments is that they are strictly for the human reader, and explicitly ignored by the tool. Taking all of these speed bumps into account, I made a new plan: A perl script reads the mapfiles, generates some small C glue code to produce empty functions and data definitions, compiles and links the stub object from the generated glue code, and then deletes the generated glue code. Another perl script used after both objects have been built, to compare the real and stub objects, using data from elfdump, and validate that they present the same linking interface. By June 2008, I had written the above, and generated a stub object for libc. It was a useful prototype process to go through, and it allowed me to explore the ideas at a deep level. Ultimately though, the result was unsatisfactory as a basis for real product. There were so many issues: The use of stylized comments were fine for a prototype, but not close to professional enough for shipping product. The idea of having to document and support it was a large concern. The ideal solution for stub objects really does involve having the link-editor accept the same arguments used to build the real object, augmented with a single extra command line option. Any other solution, such as our prototype script, will require makefiles to be modified in deeper ways to support building stubs, and so, will raise barriers to converting existing code. A validation script that rederives what the linker knew when it built an object will always be at a disadvantage relative to the actual linker that did the work. A stub object should be identifyable as such. In the prototype, there was no tag or other metadata that would let you know that they weren't real objects. Being able to identify a stub object in this way means that the file command can tell you what it is, and that the runtime linker can refuse to try and run a program that loads one. At that point, we needed to apply this prototype to building Solaris. As you might imagine, the task of modifying all the makefiles in the core Solaris code base in order to do this is a massive task, and not something you'd enter into lightly. The quality of the prototype just wasn't good enough to justify that sort of time commitment, so I tabled the project, putting it on my list of long term things to think about, and moved on to other work. It would sit there for a couple of years. Semi-coincidentally, one of the projects I tacked after that was to create a new mapfile syntax for the Solaris link-editor. We had wanted to do something about the old mapfile syntax for many years. Others before me had done some paper designs, and a great deal of thought had already gone into the features it should, and should not have, but for various reasons things had never moved beyond the idea stage. When I joined Sun in late 2005, I got involved in reviewing those things and thinking about the problem. Now in 2008, fresh from relearning for the Nth time why the old mapfile syntax was a huge impediment to linker progress, it seemed like the right time to tackle the mapfile issue. Paving the way for proper stub object support was not the driving force behind that effort, but I certainly had them in mind as I moved forward. The new mapfile syntax, which we call version 2, integrated into Nevada build snv_135 in in February 2010: 6916788 ld version 2 mapfile syntax PSARC/2009/688 Human readable and extensible ld mapfile syntax In order to prove that the new mapfile syntax was adequate for general purpose use, I had also done an overhaul of the ON consolidation to convert all mapfiles to use the new syntax, and put checks in place that would ensure that no use of the old syntax would creep back in. That work went back into snv_144 in June 2010: 6916796 OSnet mapfiles should use version 2 link-editor syntax That was a big putback, modifying 517 files, adding 18 new files, and removing 110 old ones. I would have done this putback anyway, as the work was already done, and the benefits of human readable syntax are obvious. However, among the justifications listed in CR 6916796 was this We anticipate adding additional features to the new mapfile language that will be applicable to ON, and which will require all sharable object mapfiles to use the new syntax. I never explained what those additional features were, and no one asked. It was premature to say so, but this was a reference to stub objects. By that point, I had already put together a working prototype link-editor with the necessary support for stub objects. I was pleased to find that building stubs was indeed very fast. On my desktop system (Ultra 24), an amd64 stub for libc can can be built in a fraction of a second: % ptime ld -64 -z stub -o stubs/libc.so.1 -G -hlibc.so.1 \ -ztext -zdefs -Bdirect ... real 0.019708910 user 0.010101680 sys 0.008528431 In order to go from prototype to integrated link-editor feature, I knew that I would need to prove that stub objects were valuable. And to do that, I knew that I'd have to switch the Solaris ON consolidation to use stub objects and evaluate the outcome. And in order to do that experiment, ON would first need to be converted to version 2 mapfiles. Sub-mission accomplished. Normally when you design a new feature, you can devise reasonably small tests to show it works, and then deploy it incrementally, letting it prove its value as it goes. The entire point of stub objects however was to demonstrate that they could be successfully applied to an extremely large and complex code base, and specifically to solve the Solaris build issues detailed above. There was no way to finesse the matter — in order to move ahead, I would have to successfully use stub objects to build the entire ON consolidation and demonstrate their value. In software, the need to boil the ocean can often be a warning sign that things are trending in the wrong direction. Conversely, sometimes progress demands that you build something large and new all at once. A big win, or a big loss — sometimes all you can do is try it and see what happens. And so, I spent some time staring at ON makefiles trying to get a handle on how things work, and how they'd have to change. It's a big and messy world, full of complex interactions, unspecified dependencies, special cases, and knowledge of arcane makefile features... ...and so, I backed away, put it down for a few months and did other work... ...until the fall, when I felt like it was time to stop thinking and pondering (some would say stalling) and get on with it. Without stubs, the following gives a simplified high level view of how Solaris is built: An initially empty directory known as the proto, and referenced via the ROOT makefile macro is established to receive the files that make up the Solaris distribution. A top level setup rule creates the proto area, and performs operations needed to initialize the workspace so that the main build operations can be launched, such as copying needed header files into the proto area. Parallel builds are launched to build the kernel (usr/src/uts), libraries (usr/src/lib), and commands. The install makefile target builds each item and delivers a copy to the proto area. All libraries and executables link against the objects previously installed in the proto, implying the need to synchronize the order in which things are built. Subsequent passes run lint, and do packaging. Given this structure, the additions to use stub objects are: A new second proto area is established, known as the stub proto and referenced via the STUBROOT makefile macro. The stub proto has the same structure as the real proto, but is used to hold stub objects. All files in the real proto are delivered as part of the Solaris product. In contrast, the stub proto is used to build the product, and then thrown away. A new target is added to library Makefiles called stub. This rule builds the stub objects. The ld command is designed so that you can build a stub object using the same ld command line you'd use to build the real object, with the addition of a single -z stub option. This means that the makefile rules for building the stub objects are very similar to those used to build the real objects, and many existing makefile definitions can be shared between them. A new target is added to the Makefiles called stubinstall which delivers the stub objects built by the stub rule into the stub proto. These rules reuse much of existing plumbing used by the existing install rule. The setup rule runs stubinstall over the entire lib subtree as part of its initialization. All libraries and executables link against the objects in the stub proto rather than the main proto, and can therefore be built in parallel without any synchronization. There was no small way to try this that would yield meaningful results. I would have to take a leap of faith and edit approximately 1850 makefiles and 300 mapfiles first, trusting that it would all work out. Once the editing was done, I'd type make and see what happened. This took about 6 weeks to do, and there were many dark days when I'd question the entire project, or struggle to understand some of the many twisted and complex situations I'd uncover in the makefiles. I even found a couple of new issues that required changes to the new stub object related code I'd added to ld. With a substantial amount of encouragement and help from some key people in the Solaris group, I eventually got the editing done and stub objects for the entire workspace built. I found that my desktop system could build all the stub objects in the workspace in roughly a minute. This was great news, as it meant that use of the feature is effectively free — no one was likely to notice or care about the cost of building them. After another week of typing make, fixing whatever failed, and doing it again, I succeeded in getting a complete build! The next step was to remove all of the make rules and .WAIT statements dedicated to controlling the order in which libraries under usr/src/lib are built. This came together pretty quickly, and after a few more speed bumps, I had a workspace that built cleanly and looked like something you might actually be able to integrate someday. This was a significant milestone, but there was still much left to do. I turned to doing full nightly builds. Every type of build (open, closed, OpenSolaris, export, domestic) had to be tried. Each type failed in a new and unique way, requiring some thinking and rework. As things came together, I became aware of things that could have been done better, simpler, or cleaner, and those things also required some rethinking, the seeking of wisdom from others, and some rework. After another couple of weeks, it was in close to final form. My focus turned towards the end game and integration. This was a huge workspace, and needed to go back soon, before changes in the gate would made merging increasingly difficult. At this point, I knew that the stub objects had greatly simplified the makefile logic and uncovered a number of race conditions, some of which had been there for years. I assumed that the builds were faster too, so I did some builds intended to quantify the speedup in build time that resulted from this approach. It had never occurred to me that there might not be one. And so, I was very surprised to find that the wall clock build times for a stock ON workspace were essentially identical to the times for my stub library enabled version! This is why it is important to always measure, and not just to assume. One can tell from first principles, based on all those removed dependency rules in the library makefile, that the stub object version of ON gives dmake considerably more opportunities to overlap library construction. Some hypothesis were proposed, and shot down: Could we have disabled dmakes parallel feature? No, a quick check showed things being build in parallel. It was suggested that we might be I/O bound, and so, the threads would be mostly idle. That's a plausible explanation, but system stats didn't really support it. Plus, the timing between the stub and non-stub cases were just too suspiciously identical. Are our machines already handling as much parallelism as they are capable of, and unable to exploit these additional opportunities? Once again, we didn't see the evidence to back this up. Eventually, a more plausible and obvious reason emerged: We build the libraries and commands (usr/src/lib, usr/src/cmd) in parallel with the kernel (usr/src/uts). The kernel is the long leg in that race, and so, wall clock measurements of build time are essentially showing how long it takes to build uts. Although it would have been nice to post a huge speedup immediately, we can take solace in knowing that stub objects simplify the makefiles and reduce the possibility of race conditions. The next step in reducing build time should be to find ways to reduce or overlap the uts part of the builds. When that leg of the build becomes shorter, then the increased parallelism in the libs and commands will pay additional dividends. Until then, we'll just have to settle for simpler and more robust. And so, I integrated the link-editor support for creating stub objects into snv_153 (November 2010) with 6993877 ld should produce stub objects PSARC/2010/397 ELF Stub Objects followed by the work to convert the ON consolidation in snv_161 (February 2011) with 7009826 OSnet should use stub objects 4631488 lib/Makefile is too patient: .WAITs should be reduced This was a huge putback, with 2108 modified files, 8 new files, and 2 removed files. Due to the size, I was allowed a window after snv_160 closed in which to do the putback. It went pretty smoothly for something this big, a few more preexisting race conditions would be discovered and addressed over the next few weeks, and things have been quiet since then. Conclusions and Looking Forward Solaris has been built with stub objects since February. The fact that developers no longer specify the order in which libraries are built has been a big success, and we've eliminated an entire class of build error. That's not to say that there are no build races left in the ON makefiles, but we've taken a substantial bite out of the problem while generally simplifying and improving things. The introduction of a stub proto area has also opened some interesting new possibilities for other build improvements. As this article has become quite long, and as those uses do not involve stub objects, I will defer that discussion to a future article.

Read the article

C# HashSet<T>

- by Ben Griswold

I hadn’t done much (read: anything) with the C# generic HashSet until I recently needed to produce a distinct collection. As it turns out, HashSet<T> was the perfect tool. As the following snippet demonstrates, this collection type offers a lot: // Using HashSet<T>: // http://www.albahari.com/nutshell/ch07.aspx var letters = new HashSet<char>("the quick brown fox"); Console.WriteLine(letters.Contains('t')); // true Console.WriteLine(letters.Contains('j')); // false foreach (char c in letters) Console.Write(c); // the quickbrownfx Console.WriteLine(); letters = new HashSet<char>("the quick brown fox"); letters.IntersectWith("aeiou"); foreach (char c in letters) Console.Write(c); // euio Console.WriteLine(); letters = new HashSet<char>("the quick brown fox"); letters.ExceptWith("aeiou"); foreach (char c in letters) Console.Write(c); // th qckbrwnfx Console.WriteLine(); letters = new HashSet<char>("the quick brown fox"); letters.SymmetricExceptWith("the lazy brown fox"); foreach (char c in letters) Console.Write(c); // quicklazy Console.WriteLine(); The MSDN documentation is a bit light on HashSet<T> documentation but if you search hard enough you can find some interesting information and benchmarks. But back to that distinct list I needed… // MSDN Add // http://msdn.microsoft.com/en-us/library/bb353005.aspx var employeeA = new Employee {Id = 1, Name = "Employee A"}; var employeeB = new Employee {Id = 2, Name = "Employee B"}; var employeeC = new Employee {Id = 3, Name = "Employee C"}; var employeeD = new Employee {Id = 4, Name = "Employee D"}; var naughty = new List<Employee> {employeeA}; var nice = new List<Employee> {employeeB, employeeC}; var employees = new HashSet<Employee>(); naughty.ForEach(x => employees.Add(x)); nice.ForEach(x => employees.Add(x)); foreach (Employee e in employees) Console.WriteLine(e); // Returns Employee A Employee B Employee C The Add Method returns true on success and, you guessed it, false if the item couldn’t be added to the collection. I’m using the Linq ForEach syntax to add all valid items to the employees HashSet. It works really great. This is just a rough sample, but you may have noticed I’m using Employee, a reference type. Most samples demonstrate the power of the HashSet with a collection of integers which is kind of cheating. With value types you don’t have to worry about defining your own equality members. With reference types, you do. internal class Employee { public int Id { get; set; } public string Name { get; set; } public override string ToString() { return Name; } public bool Equals(Employee other) { if (ReferenceEquals(null, other)) return false; if (ReferenceEquals(this, other)) return true; return other.Id == Id; } public override bool Equals(object obj) { if (ReferenceEquals(null, obj)) return false; if (ReferenceEquals(this, obj)) return true; if (obj.GetType() != typeof (Employee)) return false; return Equals((Employee) obj); } public override int GetHashCode() { return Id; } public static bool operator ==(Employee left, Employee right) { return Equals(left, right); } public static bool operator !=(Employee left, Employee right) { return !Equals(left, right); } } Fortunately, with Resharper, it’s a snap. Click on the class name, ALT+INS and then follow with the handy dialogues. That’s it. Try out the HashSet<T>. It’s good stuff.

Read the article

Recursively Traversing a Hierarchical Data Structure Using LINQ

Recursively traversing a hierarchical data structure using LINQ without flattenning the hierarchy or using foreach loops

Read the article

Converting a DrawModel() using BasicEffect to one using Effect

- by Fibericon

Take this DrawModel() provided by MSDN: private void DrawModel(Model m) { Matrix[] transforms = new Matrix[m.Bones.Count]; float aspectRatio = graphics.GraphicsDevice.Viewport.Width / graphics.GraphicsDevice.Viewport.Height; m.CopyAbsoluteBoneTransformsTo(transforms); Matrix projection = Matrix.CreatePerspectiveFieldOfView(MathHelper.ToRadians(45.0f), aspectRatio, 1.0f, 10000.0f); Matrix view = Matrix.CreateLookAt(new Vector3(0.0f, 50.0f, Zoom), Vector3.Zero, Vector3.Up); foreach (ModelMesh mesh in m.Meshes) { foreach (BasicEffect effect in mesh.Effects) { effect.EnableDefaultLighting(); effect.View = view; effect.Projection = projection; effect.World = gameWorldRotation * transforms[mesh.ParentBone.Index] * Matrix.CreateTranslation(Position); } mesh.Draw(); } } How would I apply a custom effect to a model with that? Effect doesn't have View, Projection, or World members. This is what they recommend replacing the foreach loop with: foreach (ModelMesh mesh in terrain.Meshes) { foreach (Effect effect in mesh.Effects) { mesh.Draw(); } } Of course, that doesn't really work. What else needs to be done?

Read the article

Rearrange array

- by bradenkeith

Read the article

What would you do if you just had this code dumped in your lap?

- by chickeninabiscuit

Man, I just had this project given to me - expand on this they say. This is an example of ONE function: <?php //500+ lines of pure wonder. function page_content_vc($content) { global $_DBH, $_TPL, $_SET; $_SET['ignoreTimezone'] = true; lu_CheckUpdateLogin(); if($_SESSION['dash']['VC']['switch'] == 'unmanned' || $_SESSION['dash']['VC']['switch'] == 'touchscreen') { if($content['page_name'] != 'vc') { header('Location: /vc/'); die(); } } if($_GET['l']) { unset($_SESSION['dash']['VC']); if($loc_id = lu_GetFieldValue('ID', 'Location', $_GET['l'])) { if(lu_CheckPermissions('vc', $loc_id)) { $timezone = lu_GetFieldValue('Time Zone', 'Location', $loc_id, 'ID'); if(strlen($timezone) > 0) { $_SESSION['time_zone'] = $timezone; } $_SESSION['dash']['VC']['loc_ID'] = $loc_id; header('Location: /vc/'); die(); } } } if($_SESSION['dash']['VC']['loc_ID']) { $timezone = lu_GetFieldValue('Time Zone', 'Location', $_SESSION['dash']['VC']['loc_ID'], 'ID'); if(strlen($timezone) > 0) { $_SESSION['time_zone'] = $timezone; } $loc_id = $_SESSION['dash']['VC']['loc_ID']; $org_id = lu_GetFieldValue('record_ID', 'Location', $loc_id); $_TPL->assign('loc_id', $loc_id); $location_name = lu_GetFieldValue('Location Name', 'Location', $loc_id); $_TPL->assign('LocationName', $location_name); $customer_name = lu_GetFieldValue('Customer Name', 'Organisation', $org_id); $_TPL->assign('CustomerName', $customer_name); $enable_visitor_snap = lu_GetFieldValue('VisitorSnap', 'Location', $loc_id); $_TPL->assign('EnableVisitorSnap', $enable_visitor_snap); $lacps = explode("\n", lu_GetFieldValue('Location Access Control Point', 'Location', $loc_id)); array_walk($lacps, 'trim_value'); if(count($lacps) > 0) { if(count($lacps) == 1) { $_SESSION['dash']['VC']['lacp'] = $lacps[0]; } else { if($_GET['changeLACP'] && in_array($_GET['changeLACP'], $lacps)) { $_SESSION['dash']['VC']['lacp'] = $_GET['changeLACP']; header('Location: /vc/'); die(); } else if(!in_array($_SESSION['dash']['VC']['lacp'], $lacps)) { $_SESSION['dash']['VC']['lacp'] = $lacps[0]; } $_TPL->assign('LACP_array', $lacps); } $_TPL->assign('current_LACP', $_SESSION['dash']['VC']['lacp']); $_TPL->assign('showContractorSearch', true); /* if($contractorStaff = lu_GetTableRow('ContractorStaff', $org_id, 'record_ID', 'record_Inactive != "checked"')) { foreach($contractorStaff['rows'] as $contractor) { $lacp_rights = lu_OrganiseCustomDataFunctionMultiselect($contractor[lu_GetFieldName('Location Access Rights', 'ContractorStaff')]); if(in_array($_SESSION['dash']['VC']['lacp'], $lacp_rights)) { $_TPL->assign('showContractorSearch', true); } } } */ } $selectedOptions = explode(',', lu_GetFieldValue('Included Fields', 'Location', $_SESSION['dash']['VC']['loc_ID'])); $newOptions = array(); foreach($selectedOptions as $selOption) { $so_array = explode('|', $selOption, 2); if(count($so_array) > 1) { $newOptions[$so_array[0]] = $so_array[1]; } else { $newOptions[$so_array[0]] = "Both"; } } if($newOptions[lu_GetFieldName('Expected Length of Visit', 'Visitor')]) { $alert = false; if($visitors = lu_OrganiseVisitors( lu_GetTableRow('Visitor', 'checked', lu_GetFieldName('Checked In', 'Visitor'), lu_GetFieldName('Location for Visit', 'Visitor').'="'.$_SESSION['dash']['VC']['loc_ID'].'" AND '.lu_GetFieldName('Checked Out', 'Visitor').' != "checked"'), false, true, true)) { foreach($visitors['rows'] as $key => $visitor) { if($visitor['expected'] && $visitor['expected'] + (60*30) < time()) { $alert = true; } } } if($alert == true) { $_TPL->assign('showAlert', 'red'); } else { //$_TPL->assign('showAlert', 'green'); } } $_TPL->assign('switch', $_SESSION['dash']['VC']['switch']); if($_SESSION['dash']['VC']['switch'] == 'touchscreen') { $_TPL->assign('VC_unmanned', true); } if($_GET['check'] == 'in') { if($_SESSION['dash']['VC']['switch'] == 'touchscreen') { lu_CheckInTouchScreen(); } else { lu_CheckIn(); } } else if($_GET['check'] == 'out') { if($_SESSION['dash']['VC']['switch'] == 'touchscreen') { lu_CheckOutTouchScreen(); } else { lu_CheckOut(); } } else if($_GET['switch'] == 'unmanned') { $_SESSION['dash']['VC']['switch'] = 'unmanned'; if($_GET['printing'] == true && (lu_GetFieldValue('Printing', 'Location', $_SESSION['dash']['VC']['loc_ID']) != "No" && lu_GetFieldValue('Printing', 'Location', $_SESSION['dash']['VC']['loc_ID']) != "")) { $_SESSION['dash']['VC']['printing'] = true; } else { $_SESSION['dash']['VC']['printing'] = false; } header('Location: /vc/'); die(); } else if($_GET['switch'] == 'touchscreen') { $_SESSION['dash']['VC']['switch'] = 'touchscreen'; if($_GET['printing'] == true && (lu_GetFieldValue('Printing', 'Location', $_SESSION['dash']['VC']['loc_ID']) != "No" && lu_GetFieldValue('Printing', 'Location', $_SESSION['dash']['VC']['loc_ID']) != "")) { $_SESSION['dash']['VC']['printing'] = true; } else { $_SESSION['dash']['VC']['printing'] = false; } header('Location: /vc/'); die(); } else if($_GET['switch'] == 'manned') { if($_POST['password']) { if(md5($_POST['password']) == $_SESSION['dash']['password']) { unset($_SESSION['dash']['VC']['switch']); //setcookie('email', "", time() - 3600); //setcookie('location', "", time() - 3600); header('Location: /vc/'); die(); } else { $_TPL->assign('switchLoginError', 'Incorrect Password'); } } $_TPL->assign('switchLogin', 'true'); } else if($_GET['m'] == 'visitor') { lu_ModifyVisitorVC(); } else if($_GET['m'] == 'enote') { lu_ModifyEnoteVC(); } else if($_GET['m'] == 'medical') { lu_ModifyMedicalVC(); } else if($_GET['print'] == 'label' && $_GET['v']) { lu_PrintLabelVC(); } else { unset($_SESSION['dash']['VC']['checkin']); unset($_SESSION['dash']['VC']['checkout']); $_TPL->assign('icon', 'GroupCheckin'); if($_SESSION['dash']['VC']['switch'] != 'unmanned' && $_SESSION['dash']['VC']['switch'] != 'touchscreen') { $staff_ids = array(); if($staffs = lu_GetTableRow('Staff', $_SESSION['dash']['VC']['loc_ID'], 'record_ID')) { foreach($staffs['rows'] as $staff) { $staff_ids[] = $staff['ID']; } } if($_GET['view'] == "tomorrow") { $dateStart = date('Y-m-d', mktime(0, 0, 0, date("m") , date("d")+1, date("Y"))); $dateEnd = date('Y-m-d', mktime(0, 0, 0, date("m") , date("d")+1, date("Y"))); } else if($_GET['view'] == "month") { $dateStart = date('Y-m-d', mktime(0, 0, 0, date("m"), date("d"), date("Y"))); $dateEnd = date('Y-m-d', mktime(0, 0, 0, date("m"), date("d")+30, date("Y"))); } else if($_GET['view'] == "week") { $dateStart = date('Y-m-d', mktime(0, 0, 0, date("m"), date("d"), date("Y"))); $dateEnd = date('Y-m-d', mktime(0, 0, 0, date("m"), date("d")+7, date("Y"))); } else { $dateStart = date('Y-m-d'); $dateEnd = date('Y-m-d'); } if(lu_GetFieldValue('Enable Survey', 'Location', $_SESSION['dash']['VC']['loc_ID']) == 'checked' && lu_GetFieldValue('Add Survey', 'Location', $_SESSION['dash']['VC']['loc_ID']) == 'checked') { $_TPL->assign('enableSurvey', true); } //lu_GetFieldName('Checked In', 'Visitor') //!= "checked" //date('d/m/Y'), lu_GetFieldName('Date of Visit', 'Visitor') if($visitors = lu_OrganiseVisitors(lu_GetTableRow('Visitor', $_SESSION['dash']['VC']['loc_ID'], lu_GetFieldName('Location for Visit', 'Visitor'), lu_GetFieldName('Checked In', 'Visitor').' != "checked" AND '.lu_GetFieldName('Checked Out', 'Visitor').' != "checked" AND '.lu_GetFieldName('Date of Visit', 'Visitor').' >= "'.$dateStart.'" AND '.lu_GetFieldName('Date of Visit', 'Visitor').' <= "'.$dateEnd.'"'))) { foreach($visitors['days'] as $day => $visitors_day) { foreach($visitors_day['rows'] as $key => $visitor) { $visitors['days'][$day]['rows'][$key]['visiting'] = lu_GetTableRow('Staff', $visitor['record_ID'], 'ID'); $visitors['days'][$day]['rows'][$key]['visiting']['notify'] = $_DBH->getRow('SELECT * FROM lu_notification WHERE ent_ID = "'.$visitor['record_ID'].'"'); } } //array_dump($visitors); $_TPL->assign('visitors', $visitors); } if($_GET['conGroup']) { if($_GET['action'] == 'add') { $_SESSION['dash']['VC']['conGroup'][$_GET['conGroup']] = $_GET['conGroup']; } else { unset($_SESSION['dash']['VC']['conGroup'][$_GET['conGroup']]); } } if(count($_SESSION['dash']['VC']['conGroup']) > 0) { if($conGroupResult = lu_GetTableRow('ContractorStaff', '1', '1', ' ID IN ('.implode(',', $_SESSION['dash']['VC']['conGroup']).')')) { if($_POST['_submit'] == 'Check-In Group >>') { $form = lu_GetForm('VisitorStandard'); $standarddata = array(); foreach($form['items'] as $key=>$item) { $standarddata[$key] = $_POST[lu_GetFieldName($item['name'], 'Visitor')]; } foreach($conGroupResult['rows'] as $conStaff) { $data = $standarddata; foreach($form['items'] as $key=>$item) { if($key != 'ID' && $key != 'record_ID' && $conStaff[lu_GetFieldName(lu_GetNameField($key, 'Visitor'), 'ContractorStaff')]) { $data[$key] = $conStaff[lu_GetFieldName(lu_GetNameField($key, 'Visitor'), 'ContractorStaff')]; } } $data['record_ID'] = $data[lu_GetFieldName('Visiting', 'Visitor')]; $data[lu_GetFieldName('Date of Visit', 'Visitor')] = date('Y-m-d'); $data[lu_GetFieldName('Time of Visit', 'Visitor')] = date('H:i'); $data[lu_GetFieldName('Checked In', 'Visitor')] = 'checked'; $data[lu_GetFieldName('Location for Visit', 'Visitor')] = $_SESSION['dash']['VC']['loc_ID']; $data[lu_GetFieldName('ConStaff ID', 'Visitor')] = $conStaff['ID']; $data[lu_GetFieldName('From', 'Visitor')] = lu_GetFieldValue('Legal Name', 'Contractor', $conStaff[lu_GetFieldName('Contractor', 'ContractorStaff')]); $id = lu_UpdateData($form, $data); lu_VisitorCheckIn($id); //array_dump($data); //array_dump($id); } unset($_SESSION['dash']['VC']['conGroup']); header('Location: /vc/'); die(); } if(count($conGroupResult['rows'])) { foreach($conGroupResult['rows'] as $key => $cstaff) { $conGroupResult['rows'][$key]['contractor'] = lu_GetTableRow('Contractor', $cstaff[lu_GetFieldName('Contractor', 'ContractorStaff')], 'ID'); } $_TPL->assign('conGroupResult', $conGroupResult); } $conGroupForm = lu_GetForm('VisitorConGroup'); $conGroupForm = lu_OrganiseVisitorForm($conGroupForm, $_SESSION['dash']['VC']['loc_ID'], 'Contractor'); $secure_options_array = lu_GetSecureOptions($org_id); if($secure_options_array[$_SESSION['dash']['VC']['loc_ID']]) { $conGroupForm['items'][lu_GetFieldName('Secure Area', 'Visitor')]['options']['values'] = $secure_options_array[$_SESSION['dash']['VC']['loc_ID']]; $conGroupForm['items'][lu_GetFieldName('Secure Area', 'Visitor')]['name'] = 'Secure Area'; } else { unset($conGroupForm['items'][lu_GetFieldName('Secure Area', 'Visitor')]); } if($secure_options_array) { $form['items'][lu_GetFieldName('Secure Area', 'Visitor')]['options']['values'] = $secure_options_array; $form['items'][lu_GetFieldName('Secure Area', 'Visitor')]['name'] = 'Secure Area'; } else { unset($form['items'][lu_GetFieldName('Secure Area', 'Visitor')]); } $_TPL->assign('conGroupForm', $conGroupForm); $_TPL->assign('hideFormCancel', true); } } if($_GET['searchVisitors']) { $_TPL->assign('searchVisitorsQuery', $_GET['searchVisitors']); $where = ''; if($_GET['searchVisitorsIn'] == 'Yes') { $where .= ' AND '.lu_GetFieldName('Checked In', 'Visitor').' = "checked"'; $_TPL->assign('searchVisitorsIn', 'Yes'); } else { $where .= ' AND '.lu_GetFieldName('Checked In', 'Visitor').' != "checked"'; $_TPL->assign('searchVisitorsIn', 'No'); } if($_GET['searchVisitorsOut'] == 'Yes') { $where = ''; $where .= ' AND '.lu_GetFieldName('Checked Out', 'Visitor').' = "checked"'; $_TPL->assign('searchVisitorsOut', 'Yes'); } else { $where .= ' AND '.lu_GetFieldName('Checked Out', 'Visitor').' != "checked"'; $_TPL->assign('searchVisitorsOut', 'No'); } if($searchVisitors = lu_OrganiseVisitors(lu_GetTableRow('Visitor', $_GET['searchVisitors'], '#search#', lu_GetFieldName('Location for Visit', 'Visitor').'="'.$_SESSION['dash']['VC']['loc_ID'].'"'.$where))) { foreach($searchVisitors['rows'] as $key => $visitor) { $searchVisitors['rows'][$key]['visiting'] = lu_GetTableRow('Staff', $visitor['record_ID'], 'ID'); } $_TPL->assign('searchVisitors', $searchVisitors); } else { $_TPL->assign('searchVisitorsNotFound', true); } } else if($_GET['searchStaff']) { if($_POST['staff_id']) { if(lu_CheckPermissions('staff', $_POST['staff_id'])) { $_DBH->query('UPDATE '.lu_GetTableName('Staff').' SET '.lu_GetFieldName('Current Location', 'Staff').' = "'.$_POST['current_location'].'" WHERE ID="'.$_POST['staff_id'].'"'); } } $locations = lu_GetTableRow('Location', $org_id, 'record_ID'); if(count($locations['rows']) > 1) { $_TPL->assign('staffLocations', $locations); } $loc_ids = array(); foreach($locations['rows'] as $location) { $loc_ids[] = $location['ID']; } // array_dump($locations); // array_dump($_POST); $_TPL->assign('searchStaffQuery', $_GET['searchStaff']); $where = ' AND record_Inactive != "checked"'; if($_GET['searchStaffIn'] == 'Yes' && $_GET['searchStaffOut'] != 'Yes') { $where .= ' AND ('.lu_GetFieldName('Staff Status', 'Staff').' = "" OR '.lu_GetFieldName('Staff Status', 'Staff').' = "On-Site")'. $_TPL->assign('searchStaffIn', 'Yes'); $_TPL->assign('searchStaffOut', 'No'); } else if($_GET['searchStaffOut'] == 'Yes' && $_GET['searchStaffIn'] != 'Yes') { $where .= ' AND ('.lu_GetFieldName('Staff Status', 'Staff').' != "" AND '.lu_GetFieldName('Staff Status', 'Staff').' != "On-Site")'. $_TPL->assign('searchStaffOut', 'Yes'); $_TPL->assign('searchStaffIn', 'No'); } else { $_TPL->assign('searchStaffOut', 'Yes'); $_TPL->assign('searchStaffIn', 'Yes'); } if($searchStaffs = lu_GetTableRow('Staff', $_GET['searchStaff'], '#search#', 'record_ID IN ('.implode(',', $loc_ids).')'.$where, lu_GetFieldName('First Name', 'Staff').','.lu_GetFieldName('Surname', 'Staff'))) { $_TPL->assign('searchStaffs', $searchStaffs); } else { $_TPL->assign('searchStaffNotFound', true); } } else if($_GET['searchContractor']) { $_TPL->assign('searchContractorQuery', $_GET['searchContractor']); //$where = ' AND '.lu_GetTableName('ContractorStaff').'.record_Inactive != "checked"'; $where = ' '; if($_GET['searchContractorIn'] == 'Yes' && $_GET['searchContractorOut'] != 'Yes') { $where .= ' AND ('.lu_GetFieldName('Onsite Status', 'ContractorStaff').' = "Onsite")'; $_TPL->assign('searchContractorIn', 'Yes'); $_TPL->assign('searchContractorOut', 'No'); } else if($_GET['searchContractorOut'] == 'Yes' && $_GET['searchContractorIn'] != 'Yes') { $where .= ' AND ('.lu_GetFieldName('Onsite Status', 'ContractorStaff').' != "Onsite")'. $_TPL->assign('searchContractorOut', 'Yes'); $_TPL->assign('searchContractorIn', 'No'); } else { $_TPL->assign('searchContractorOut', 'Yes'); $_TPL->assign('searchContractorIn', 'Yes'); } $join = 'LEFT JOIN '.lu_GetTableName('Contractor').' ON '.lu_GetTableName('Contractor').'.ID = '.lu_GetTableName('ContractorStaff').'.'.lu_GetFieldName('Contractor', 'ContractorStaff'); $extrasearch = array ( lu_GetTableName('Contractor').'.'.lu_GetFieldName('Legal Name', 'Contractor') ); if($searchContractorResult = lu_GetTableRow('ContractorStaff', $_GET['searchContractor'], '#search#', lu_GetTableName('ContractorStaff').'.record_ID = "'.$org_id.'" '.$where, lu_GetFieldName('First Name', 'ContractorStaff').','.lu_GetFieldName('Surname', 'ContractorStaff'), $join, $extrasearch)) { /* foreach($searchContractorResult['rows'] as $key=>$contractor) { $lacp_rights = lu_OrganiseCustomDataFunctionMultiselect($contractor[lu_GetFieldName('Location Access Rights', 'ContractorStaff')]); if(!in_array($_SESSION['dash']['VC']['lacp'], $lacp_rights)) { unset($searchContractorResult['rows'][$key]); } } */ if(count($searchContractorResult['rows'])) { foreach($searchContractorResult['rows'] as $key => $cstaff) { /* if($cstaff[lu_GetFieldName('Onsite_Status', 'Contractor')] == 'Onsite')) { if($visitor['rows'][0][lu_GetFieldName('ConStaff ID', 'Visitor')]) { $_DBH->query('UPDATE '.lu_GetTableName('ContractorStaff').' SET '.lu_GetFieldName('Onsite Status', 'ContractorStaff').' = "" WHERE ID="'.$visitor['rows'][0][lu_GetFieldName('ConStaff ID', 'Visitor')].'"'); } } */ if($cstaff[lu_GetFieldName('SACN Expiry Date', 'ContractorStaff')] != '0000-00-00') { if(strtotime($cstaff[lu_GetFieldName('SACN Expiry Date', 'ContractorStaff')]) < time()) { $searchContractorResult['rows'][$key]['sacn_expiry'] = true; } else { $searchContractorResult['rows'][$key]['sacn_expiry'] = false; } } else { $searchContractorResult['rows'][$key]['sacn_expiry'] = false; } if($cstaff[lu_GetFieldName('Induction Valid Until', 'ContractorStaff')] != '0000-00-00') { if(strtotime($cstaff[lu_GetFieldName('Induction Valid Until', 'ContractorStaff')]) < time()) { $searchContractorResult['rows'][$key]['induction_expiry'] = true; } else { $searchContractorResult['rows'][$key]['induction_expiry'] = false; } } else { $searchContractorResult['rows'][$key]['induction_expiry'] = false; } $searchContractorResult['rows'][$key]['contractor'] = lu_GetTableRow('Contractor', $cstaff[lu_GetFieldName('Contractor', 'ContractorStaff')], 'ID'); } $_TPL->assign('searchContractorResult', $searchContractorResult); } else { $_TPL->assign('searchContractorNotFound', true); } } else { $_TPL->assign('searchContractorNotFound', true); } } $occupancy = array(); $occupancy['staffNumber'] = $_DBH->getOne('SELECT count(*) FROM '.lu_GetTableName('Staff').' WHERE record_ID = "'.$_SESSION['dash']['VC']['loc_ID'].'" AND record_Inactive != "checked" AND '.lu_GetFieldName('Ignore Counts', 'Staff').' != "checked"'); $occupancy['staffNumberOnsite']= $_DBH->getOne( 'SELECT count(*) FROM '.lu_GetTableName('Staff').' WHERE ( (record_ID = "'.$_SESSION['dash']['VC']['loc_ID'].'" AND ('.lu_GetFieldName('Staff Status', 'Staff').' = "" OR '.lu_GetFieldName('Staff Status', 'Staff').' = "On-Site")) OR '.lu_GetFieldName('Current Location', 'Staff').' = "'.$_SESSION['dash']['VC']['loc_ID'].'") AND record_Inactive != "checked" AND '.lu_GetFieldName('Ignore Counts', 'Staff').' != "checked"'); $occupancy['visitorsOnsite'] = $_DBH->getOne('SELECT count(*) FROM '.lu_GetTableName('Visitor').' WHERE '.lu_GetFieldName('Location for Visit', 'Visitor').' = "'.$_SESSION['dash']['VC']['loc_ID'].'" AND '.lu_GetFieldName('Checked In', 'Visitor').' = "checked" AND '.lu_GetFieldName('Checked Out', 'Visitor').' != "checked"'); $_TPL->assign('occupancy', $occupancy); if($enotes = lu_GetTableRow('Enote', $org_id, 'record_ID', lu_GetFieldName('Note Emailed', 'Enote').' = "0000-00-00" AND '.lu_GetFieldName('Note Passed On', 'Enote').' != "Yes"')) { $_TPL->assign('EnoteNotice', true); } if($medical = lu_GetTableRow('MedicalRoom', $_SESSION['dash']['VC']['loc_ID'], 'record_ID', 'record_Inactive != "Yes"')) { $_TPL->assign('MedicalNotice', true); } if(lu_GetFieldValue('Printing', 'Location', $_SESSION['dash']['VC']['loc_ID']) != "No" && lu_GetFieldValue('Printing', 'Location', $_SESSION['dash']['VC']['loc_ID']) != "") { $_TPL->assign('UnmannedPrinting', true); } } else { if($_SESSION['dash']['VC']['printing'] == true) { $_TPL->assign('UnmannedPrinting', true); } } // enable if contractor check-in buttons should be enabled if(lu_GetFieldValue('Enable Contractor Check In', 'Location', $_SESSION['dash']['VC']['loc_ID']) == "checked") { $_TPL->assign('ContractorCheckin', true); } } if($_SESSION['dash']['entity_id'] && $_GET['fixupCon'] == 'true') { $conStaffs = lu_GetTableRow('ContractorStaff', $_SESSION['dash']['ModifyConStaffs']['org_ID'], 'record_ID', '', lu_GetFieldName('First Name', 'ContractorStaff').','.lu_GetFieldName('Surname', 'ContractorStaff')); foreach($conStaffs['rows'] as $key => $cstaff) { if($cstaff[lu_GetFieldName('Site Access Card Number', 'ContractorStaff')] && $cstaff[lu_GetFieldName('Site Access Card Type', 'ContractorStaff')]) { echo $cstaff['ID'].' '; $_DBH->query('UPDATE '.lu_GetTableName('Visitor').' SET '.lu_GetFieldName('Site Access Card Number', 'Visitor').' = "'.$cstaff[lu_GetFieldName('Site Access Card Number', 'ContractorStaff')].'", '.lu_GetFieldName('Site Access Card Type', 'Visitor').' = "'.$cstaff[lu_GetFieldName('Site Access Card Type', 'ContractorStaff')].'" WHERE '.lu_GetFieldName('ConStaff ID', 'Visitor').'="'.$cstaff['ID'].'"'); } } } } else { if($_SESSION['dash']['staffs']) { foreach($_SESSION['dash']['staffs']['rows'] as $staff) { if($staff[lu_GetFieldName('Reception Manager', 'Staff')] == 'checked') { $loc_id = $staff['record_ID']; unset($_SESSION['dash']['VC']); if($loc_id = lu_GetFieldValue('ID', 'Location', $loc_id)) { $_SESSION['dash']['VC']['loc_ID'] = $loc_id; header('Location: /vc/'); die(); } } } } $_TPL->assign('mode', 'public'); } $content['page_content'] = $_TPL->fetch('modules/vc.htm'); return $content; } ?> die();die();die();die();die(); This question will probably be closed - i just need some support from my coding brothers and sisters. *SOB*

Read the article

Tomcat 7 taking ages to start up after upgrade

- by Lawrence

I recently updated my server installation from Tomcat 6 to Tomcat 7, in order to take advantage of better connection pooling. My project uses Hibernate, for object persistance, a Mysql 5.5.20 database, and memcached for caching. When I was using Tomcat 6, Tomcat would start in about 8 seconds. After moving to Tomcat 7, it now takes between 75 - 80 seconds to start (this is on a Macbook pro 15", core i7 2Ghz, 8Gb of RAM). The only thing that has really changed between during the move from Tomcat 6 to 7 has been my context.xml file, which controls the connection pooling information: <Context antiJARLocking="true" reloadable="true" path=""> <Resource name="jdbc/test-db" auth="Container" type="javax.sql.DataSource" factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" testOnBorrow="true" testOnReturn="false" testWhileIdle="true" validationQuery="SELECT 1" validationQueryTimeout="20000" validationInterval="30000" timeBetweenEvictionRunsMillis="60000" logValidationErrors="true" autoReconnect="true" username="webuser" password="xxxxxxx" driverClassName="com.mysql.jdbc.Driver" url="jdbc:mysql://databasename.us-east-1.rds.amazonaws.com:3306/test-db" maxActive="15" minIdle="2" maxIdle="10" maxWait="10000" maxAge="7200000"/> </Context> Now, as you can see, the database is running on Amazon RDS (where our live servers are), and thus is about 200ms round trip time away from my machine. I have already checked that I have security permissions to that database from my machine, (and anyway, it connects after 75 secs, so it cant be that). My initial thought was that Tomcat 7 and hibernate are doing something weird (like pre-instantiating a bunch of connections or something), and the latency to the database is amplifying the effects. While trying to diagnose the problem, I used jstack to get a stack trace of the Tomcat 7 server while its doing its startup thing. Here is the stack trace... Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.12-b01-434 mixed mode): "Attach Listener" daemon prio=9 tid=7fa4c0038800 nid=0x10c39a000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Abandoned connection cleanup thread" daemon prio=5 tid=7fa4bb810000 nid=0x10f3ba000 in Object.wait() [10f3b9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f40a0070> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <7f40a0070> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at com.mysql.jdbc.NonRegisteringDriver$1.run(NonRegisteringDriver.java:93) "PoolCleaner[545768040:1352724902327]" daemon prio=5 tid=7fa4be852800 nid=0x10e772000 in Object.wait() [10e771000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f40c7c90> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <7f40c7c90> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "localhost-startStop-1" daemon prio=5 tid=7fa4bd034800 nid=0x10d66b000 runnable [10d668000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114) at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161) at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189) - locked <7f3673be0> (a com.mysql.jdbc.util.ReadAheadInputStream) at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3014) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3467) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3997) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2713) - locked <7f366a1c0> (a com.mysql.jdbc.JDBC4Connection) at com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1930) at com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3571) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2445) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215) - locked <7f366a1c0> (a com.mysql.jdbc.JDBC4Connection) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor10.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334) at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:699) at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:631) at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:485) at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:143) at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:116) - locked <7f34f0dc8> (a org.apache.tomcat.jdbc.pool.DataSource) at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:103) at org.apache.tomcat.jdbc.pool.DataSourceFactory.createDataSource(DataSourceFactory.java:539) at org.apache.tomcat.jdbc.pool.DataSourceFactory.getObjectInstance(DataSourceFactory.java:237) at org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:143) at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304) at org.apache.naming.NamingContext.lookup(NamingContext.java:843) at org.apache.naming.NamingContext.lookup(NamingContext.java:154) at org.apache.naming.NamingContext.lookup(NamingContext.java:831) at org.apache.naming.NamingContext.lookup(NamingContext.java:168) at org.apache.catalina.core.NamingContextListener.addResource(NamingContextListener.java:1061) at org.apache.catalina.core.NamingContextListener.createNamingContext(NamingContextListener.java:671) at org.apache.catalina.core.NamingContextListener.lifecycleEvent(NamingContextListener.java:270) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5173) - locked <7f46b07f0> (a org.apache.catalina.core.StandardContext) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) - locked <7f46b07f0> (a org.apache.catalina.core.StandardContext) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) "Catalina-startStop-1" daemon prio=5 tid=7fa4b7a5e800 nid=0x10d568000 waiting on condition [10d567000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f480e970> (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1123) - locked <7f453c630> (a org.apache.catalina.core.StandardHost) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:800) - locked <7f453c630> (a org.apache.catalina.core.StandardHost) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) - locked <7f453c630> (a org.apache.catalina.core.StandardHost) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) "GC Daemon" daemon prio=2 tid=7fa4b9912800 nid=0x10d465000 in Object.wait() [10d464000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f4506d28> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:100) - locked <7f4506d28> (a sun.misc.GC$LatencyLock) "Low Memory Detector" daemon prio=5 tid=7fa4b480b800 nid=0x10c8ae000 runnable [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=9 tid=7fa4b480b000 nid=0x10c7ab000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=9 tid=7fa4b480a000 nid=0x10c6a8000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=9 tid=7fa4b4809800 nid=0x10c5a5000 runnable [00000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fa4b4808800 nid=0x10c4a2000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=7fa4b793f000 nid=0x10c297000 in Object.wait() [10c296000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f451c8f0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <7f451c8f0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=7fa4b793e000 nid=0x10c194000 in Object.wait() [10c193000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f452e168> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <7f452e168> (a java.lang.ref.Reference$Lock) "main" prio=5 tid=7fa4b7800800 nid=0x104329000 waiting on condition [104327000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <7f480e9a0> (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1123) - locked <7f451fd90> (a org.apache.catalina.core.StandardEngine) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:302) - locked <7f451fd90> (a org.apache.catalina.core.StandardEngine) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) - locked <7f451fd90> (a org.apache.catalina.core.StandardEngine) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:443) - locked <7f451fd90> (a org.apache.catalina.core.StandardEngine) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) - locked <7f453e810> (a org.apache.catalina.core.StandardService) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:732) - locked <7f4506d58> (a [Lorg.apache.catalina.Service;) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) - locked <7f44f7ba0> (a org.apache.catalina.core.StandardServer) at org.apache.catalina.startup.Catalina.start(Catalina.java:684) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:451) "VM Thread" prio=9 tid=7fa4b7939800 nid=0x10c091000 runnable "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fa4b7802000 nid=0x10772b000 runnable "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fa4b7802800 nid=0x10782e000 runnable "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fa4b7803000 nid=0x107931000 runnable "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fa4b7804000 nid=0x107a34000 runnable "Gang worker#4 (Parallel GC Threads)" prio=9 tid=7fa4b7804800 nid=0x107b37000 runnable "Gang worker#5 (Parallel GC Threads)" prio=9 tid=7fa4b7805000 nid=0x107c3a000 runnable "Gang worker#6 (Parallel GC Threads)" prio=9 tid=7fa4b7805800 nid=0x107d3d000 runnable "Gang worker#7 (Parallel GC Threads)" prio=9 tid=7fa4b7806800 nid=0x107e40000 runnable "Concurrent Mark-Sweep GC Thread" prio=9 tid=7fa4b78e3800 nid=0x10bd0b000 runnable "Gang worker#0 (Parallel CMS Threads)" prio=9 tid=7fa4b78e2800 nid=0x10b305000 runnable "Gang worker#1 (Parallel CMS Threads)" prio=9 tid=7fa4b78e3000 nid=0x10b408000 runnable "VM Periodic Task Thread" prio=10 tid=7fa4b4815800 nid=0x10c9b1000 waiting on condition "Exception Catcher Thread" prio=10 tid=7fa4b7801800 nid=0x104554000 runnable JNI global references: 919 The only thing I can figure out from this is that it looks like the mysql jdbc drivers might have something to do with the long start up (the various stack traces I took during the start up process all pretty much look the same as this). Could anyone shed some light on what might be causing this? Have I done something dense in my context.xml? Is hibernate perhaps to blame?

Read the article

Convert ddply {plyr} to Oracle R Enterprise, or use with Embedded R Execution

- by Mark Hornick

The plyr package contains a set of tools for partitioning a problem into smaller sub-problems that can be more easily processed. One function within {plyr} is ddply, which allows you to specify subsets of a data.frame and then apply a function to each subset. The result is gathered into a single data.frame. Such a capability is very convenient. The function ddply also has a parallel option that if TRUE, will apply the function in parallel, using the backend provided by foreach. This type of functionality is available through Oracle R Enterprise using the ore.groupApply function. In this blog post, we show a few examples from Sean Anderson's "A quick introduction to plyr" to illustrate the correpsonding functionality using ore.groupApply. To get started, we'll create a demo data set and load the plyr package. set.seed(1) d <- data.frame(year = rep(2000:2014, each = 3), count = round(runif(45, 0, 20))) dim(d) library(plyr) This first example takes the data frame, partitions it by year, and calculates the coefficient of variation of the count, returning a data frame. # Example 1 res <- ddply(d, "year", function(x) { mean.count <- mean(x$count) sd.count <- sd(x$count) cv <- sd.count/mean.count data.frame(cv.count = cv) }) To illustrate the equivalent functionality in Oracle R Enterprise, using embedded R execution, we use the ore.groupApply function on the same data, but pushed to the database, creating an ore.frame. The function ore.push creates a temporary table in the database, returning a proxy object, the ore.frame. D <- ore.push(d) res <- ore.groupApply (D, D$year, function(x) { mean.count <- mean(x$count) sd.count <- sd(x$count) cv <- sd.count/mean.count data.frame(year=x$year[1], cv.count = cv) }, FUN.VALUE=data.frame(year=1, cv.count=1)) You'll notice the similarities in the first three arguments. With ore.groupApply, we augment the function to return the specific data.frame we want. We also specify the argument FUN.VALUE, which describes the resulting data.frame. From our previous blog posts, you may recall that by default, ore.groupApply returns an ore.list containing the results of each function invocation. To get a data.frame, we specify the structure of the result. The results in both cases are the same, however the ore.groupApply result is an ore.frame. In this case the data stays in the database until it's actually required. This can result in significant memory and time savings whe data is large. R> class(res) [1] "ore.frame" attr(,"package") [1] "OREbase" R> head(res) year cv.count 1 2000 0.3984848 2 2001 0.6062178 3 2002 0.2309401 4 2003 0.5773503 5 2004 0.3069680 6 2005 0.3431743 To make the ore.groupApply execute in parallel, you can specify the argument parallel with either TRUE, to use default database parallelism, or to a specific number, which serves as a hint to the database as to how many parallel R engines should be used. The next ddply example uses the summarise function, which creates a new data.frame. In ore.groupApply, the year column is passed in with the data. Since no automatic creation of columns takes place, we explicitly set the year column in the data.frame result to the value of the first row, since all rows received by the function have the same year. # Example 2 ddply(d, "year", summarise, mean.count = mean(count)) res <- ore.groupApply (D, D$year, function(x) { mean.count <- mean(x$count) data.frame(year=x$year[1], mean.count = mean.count) }, FUN.VALUE=data.frame(year=1, mean.count=1)) R> head(res) year mean.count 1 2000 7.666667 2 2001 13.333333 3 2002 15.000000 4 2003 3.000000 5 2004 12.333333 6 2005 14.666667 Example 3 uses the transform function with ddply, which modifies the existing data.frame. With ore.groupApply, we again construct the data.frame explicilty, which is returned as an ore.frame. # Example 3 ddply(d, "year", transform, total.count = sum(count)) res <- ore.groupApply (D, D$year, function(x) { total.count <- sum(x$count) data.frame(year=x$year[1], count=x$count, total.count = total.count) }, FUN.VALUE=data.frame(year=1, count=1, total.count=1)) > head(res) year count total.count 1 2000 5 23 2 2000 7 23 3 2000 11 23 4 2001 18 40 5 2001 4 40 6 2001 18 40 In Example 4, the mutate function with ddply enables you to define new columns that build on columns just defined. Since the construction of the data.frame using ore.groupApply is explicit, you always have complete control over when and how to use columns. # Example 4 ddply(d, "year", mutate, mu = mean(count), sigma = sd(count), cv = sigma/mu) res <- ore.groupApply (D, D$year, function(x) { mu <- mean(x$count) sigma <- sd(x$count) cv <- sigma/mu data.frame(year=x$year[1], count=x$count, mu=mu, sigma=sigma, cv=cv) }, FUN.VALUE=data.frame(year=1, count=1, mu=1,sigma=1,cv=1)) R> head(res) year count mu sigma cv 1 2000 5 7.666667 3.055050 0.3984848 2 2000 7 7.666667 3.055050 0.3984848 3 2000 11 7.666667 3.055050 0.3984848 4 2001 18 13.333333 8.082904 0.6062178 5 2001 4 13.333333 8.082904 0.6062178 6 2001 18 13.333333 8.082904 0.6062178 In Example 5, ddply is used to partition data on multiple columns before constructing the result. Realizing this with ore.groupApply involves creating an index column out of the concatenation of the columns used for partitioning. This example also allows us to illustrate using the ORE transparency layer to subset the data. # Example 5 baseball.dat <- subset(baseball, year > 2000) # data from the plyr package x <- ddply(baseball.dat, c("year", "team"), summarize, homeruns = sum(hr)) We first push the data set to the database to get an ore.frame. We then add the composite column and perform the subset, using the transparency layer. Since the results from database execution are unordered, we will explicitly sort these results and view the first 6 rows. BB.DAT <- ore.push(baseball) BB.DAT$index <- with(BB.DAT, paste(year, team, sep="+")) BB.DAT2 <- subset(BB.DAT, year > 2000) X <- ore.groupApply (BB.DAT2, BB.DAT2$index, function(x) { data.frame(year=x$year[1], team=x$team[1], homeruns=sum(x$hr)) }, FUN.VALUE=data.frame(year=1, team="A", homeruns=1), parallel=FALSE) res <- ore.sort(X, by=c("year","team")) R> head(res) year team homeruns 1 2001 ANA 4 2 2001 ARI 155 3 2001 ATL 63 4 2001 BAL 58 5 2001 BOS 77 6 2001 CHA 63 Our next example is derived from the ggplot function documentation. This illustrates the use of ddply within using the ggplot2 package. We first create a data.frame with demo data and use ddply to create some statistics for each group (gp). We then use ggplot to produce the graph. We can take this same code, push the data.frame df to the database and invoke this on the database server. The graph will be returned to the client window, as depicted below. # Example 6 with ggplot2 library(ggplot2) df <- data.frame(gp = factor(rep(letters[1:3], each = 10)), y = rnorm(30)) # Compute sample mean and standard deviation in each group library(plyr) ds <- ddply(df, .(gp), summarise, mean = mean(y), sd = sd(y)) # Set up a skeleton ggplot object and add layers: ggplot() + geom_point(data = df, aes(x = gp, y = y)) + geom_point(data = ds, aes(x = gp, y = mean), colour = 'red', size = 3) + geom_errorbar(data = ds, aes(x = gp, y = mean, ymin = mean - sd, ymax = mean + sd), colour = 'red', width = 0.4) DF <- ore.push(df) ore.tableApply(DF, function(df) { library(ggplot2) library(plyr) ds <- ddply(df, .(gp), summarise, mean = mean(y), sd = sd(y)) ggplot() + geom_point(data = df, aes(x = gp, y = y)) + geom_point(data = ds, aes(x = gp, y = mean), colour = 'red', size = 3) + geom_errorbar(data = ds, aes(x = gp, y = mean, ymin = mean - sd, ymax = mean + sd), colour = 'red', width = 0.4) }) But let's take this one step further. Suppose we wanted to produce multiple graphs, partitioned on some index column. We replicate the data three times and add some noise to the y values, just to make the graphs a little different. We also create an index column to form our three partitions. Note that we've also specified that this should be executed in parallel, allowing Oracle Database to control and manage the server-side R engines. The result of ore.groupApply is an ore.list that contains the three graphs. Each graph can be viewed by printing the list element. df2 <- rbind(df,df,df) df2$y <- df2$y + rnorm(nrow(df2)) df2$index <- c(rep(1,300), rep(2,300), rep(3,300)) DF2 <- ore.push(df2) res <- ore.groupApply(DF2, DF2$index, function(df) { df <- df[,1:2] library(ggplot2) library(plyr) ds <- ddply(df, .(gp), summarise, mean = mean(y), sd = sd(y)) ggplot() + geom_point(data = df, aes(x = gp, y = y)) + geom_point(data = ds, aes(x = gp, y = mean), colour = 'red', size = 3) + geom_errorbar(data = ds, aes(x = gp, y = mean, ymin = mean - sd, ymax = mean + sd), colour = 'red', width = 0.4) }, parallel=TRUE) res[[1]] res[[2]] res[[3]] To recap, we've illustrated how various uses of ddply from the plyr package can be realized in ore.groupApply, which affords the user explicit control over the contents of the data.frame result in a straightforward manner. We've also highlighted how ddply can be used within an ore.groupApply call.

Search Results

Search found 6559 results on 263 pages for 'parallel foreach'.

Page 36/263 | < Previous Page | 32 33 34 35 36 37 38 39 40 41 42 43 | Next Page >

- by alex

- by Shiraz Bhaiji

- by John

- by Randy Minder

- by Eugene

- by Kostas

- by Pmarcoen

- by Zombies

- by Dr. Zim

- by Dutch

- by Eric

- by Dutch

- by Jay

- by Beti Chode

- by melling

- by tstenner

- by user9154181

- by Ben Griswold

- by Fibericon

- by bradenkeith

- by chickeninabiscuit

- by Lawrence

- by Mark Hornick

< Previous Page | 32 33 34 35 36 37 38 39 40 41 42 43 | Next Page >