windows7 32 - Page 287 - Developer IT

WCF Authentication on the Internet - HELP

- by Eddie

I have a WCF service using the basicHTTP binding. The service will be targeted to be deployed in production in a DMZ environment on a Windows Server 2008 64 bit running IIS 7.0 and is not in an Active Directory domain. The service will be accessed by a business partner over the Internet with SSL protection. Originally, I had built the service to use x.509 Message authentication with wsHTTPBinding and after a lot of problems I punted and decided to back up and use basicHTTP with UserName authentication. Result: same exact, obscure error message as I received with certificate mode. The service works perfectly inside our domain with the exact same authentication but as soon as I move it to the DMZ I get an error reading: "An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail". The inner exception message is: "An error occurred when verifying security for the message." The services' web config with binding configuration is as follows: <services> <service behaviorConfiguration="HSSanoviaFacade.Service1Behavior" name="HSSanoviaFacade.HSSanoviaFacade"> <endpoint address="" binding="basicHttpBinding" contract="HSSanoviaFacade.IHSSanoviaFacade" bindingConfiguration="basicHttp"> <identity> <dns value="localhost" /> </identity> </endpoint> <endpoint address="mex" binding="mexHttpsBinding" contract="IMetadataExchange" /> <host> <baseAddresses> <add baseAddress="https://FULLY QUALIFIED HOST NAME CHANGED TO PROTECT/> </baseAddresses> </host> </service> </services> <bindings> <basicHttpBinding> <binding name="basicHttp"> <security mode="TransportWithMessageCredential"> <message clientCredentialType="UserName" /> </security> </binding> </basicHttpBinding> </bindings> <behaviors> <serviceBehaviors> <behavior name="HSSanoviaFacade.Service1Behavior"> <serviceMetadata httpsGetEnabled="True" /> <serviceDebug includeExceptionDetailInFaults="True" /> </behavior> </serviceBehaviors> </behaviors> The test client's configuration that gets the error: <bindings> <basicHttpBinding> <binding name="BasicHttpBinding_IHSSanoviaFacade" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true"> <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" /> <security mode="TransportWithMessageCredential"> <transport clientCredentialType="None" proxyCredentialType="None" realm="" /> <message clientCredentialType="UserName" algorithmSuite="Default" /> </security> </binding> </basicHttpBinding> </bindings> <client> <endpoint address="https://HOST NAME CHANGED TO PROTECT" binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IHSSanoviaFacade" contract="MembersService.IHSSanoviaFacade" name="BasicHttpBinding_IHSSanoviaFacade" /> </client> As mentioned earlier, the service works perfectly on the domain and the production IIS box is not on a domain. I have been tweaking and pulling my hair out for 2 weeks now and nothing seems to work. If anyone can help I would appreciate it. Even a recommendation for a work around for authentication. I'd rather not use a custom authentication scheme but use built-in SOAP capabilities. The credentials pass in thru the proxy i.e. proxy.ClientCredentials.UserName.UserName and proxy.ClientCredentials.UserName.Password are valid accounts on both the internal domain in the test environment and as a machine account on the DMZ IIS box.

Read the article

Filemaker Pro 9 (Mac) : How do I get it to deal with absolute paths?

- by Bernd Haug

I have an installation where FM Pro 9 clients open a solution from an FM Server 9. This solution then needs to access files on a network share from the clients. So far, the network share was mounted with AFP, but an infrastructure change required it to be switched to static NFS mounts. Their boot Volumes may have different names, but they all mount an NFS share at the same mount point in the "real" mount tree (starting from the UNIX root dir, /). According to http://www.filemaker.com/help/html/create_db.8.32.html#1030283 it looks like there is no way to just use a full path without having a volume name as if this was Mac OS classic - is there some way to work around this? Upgrading to a newer FileMaker is not a sought solution.

Read the article

PROBLEM: PHP strip_tags & multi-dimensional array form parameter

- by Tunji Gbadamosi

I'm having problems stripping the tags from the textual inputs retrieved from my form so as to do something with them in checkout.php. The input is stored in a multi-dimensional array. Here's my form: echo '<form name="choose" action="checkout.php" method="post" onsubmit="return validate_second_form(this);">'; echo '<input type="hidden" name="hidden_value" value="'.$no_guests.'" />'; if($no_guests >= 1){ echo '<div class="volunteer">'; echo '<fieldset>'; echo '<legend>Volunteer:</legend>'; echo '<label>Table:</label>'; echo '<select name="volunteer_table">'; foreach($tables as $t){ echo '<option>'.$t.'</option>'; } echo '</select><br><br>'; echo '<label>Seat number:</label>'; echo '<select name="volunteer_seat">'; foreach($seats as $seat){ echo '<option>'.$seat.'</option>'; } echo '</select><br><br>'; //echo '<br>'; echo '</fieldset>'; echo '</div>'; for($i=0;$i<$no_guests;$i++){ $guest = "guest_".$i; echo '<div class="'.$guest.'">'; echo '<fieldset>'; echo '<legend>Guest '.$i.':</legend>'; echo '<label>First Name:</label>'; echo '<input type="text" name="guest['.$i.']['.$first_name.']" id="fn'.$i.'">'; echo '<label>Surname:</label>'; echo '<input type="text" name="guest['.$i.']['.$surname.']" id="surname'.$i.'"><br><br>'; echo '<label>Date of Birth:</label> <br>'; echo '<label>Day:</label>'; echo '<select name="guest['.$i.'][dob_day]">'; for($j=1;$j<32;$j++){ echo"<option value='$j'>$j</option>"; } echo '</select>'; echo '<label>Month:</label>'; echo '<select name="guest['.$i.'][dob_month]">'; for($j=0;$j<sizeof($month);$j++){ $value = ($j + 1); echo"<option value='$value'>$month[$j]</option>"; } echo '</select>'; echo '<label>Year:</label>'; echo '<select name="guest['.$i.'][dob_year]">'; for($j=1900;$j<$year_limit;$j++){ echo"<option value='$j'>$j</option>"; } echo '</select> <br><br>'; echo '<label>Sex:</label>'; echo '<select name="guest['.$i.']['.$sex.']">'; echo '<option>Female</option>'; echo '<option>Male</option>'; echo '</select><br><br>'; echo '<label>Table:</label>'; echo '<select name="guest['.$i.']['.$table.']">'; foreach($tables as $t){ echo '<option>'.$t.'</option>'; } echo '</select><br><br>'; echo '<label>Seat number:</label>'; echo '<select name="guest['.$i.']['.$seat_no.']">'; foreach($seats as $seat){ echo '<option>'.$seat.'</option>'; } echo '</select><br><br>'; //echo '<br>'; echo '</fieldset>'; echo '</div>'; } } else{ echo '<div id="volunteer">'; echo '<fieldset>'; echo '<legend>Volunteer:</legend>'; echo '<label>Table:</label>'; echo '<select name="volunteer['.$table.']">'; foreach($tables as $t){ echo '<option>'.$t.'</option>'; } echo '</select><br><br>'; echo '<label>Seat number:</label>'; echo '<select name="volunteer['.$seat_no.']">'; foreach($seats as $seat){ echo '<option>'.$seat.'</option>'; } echo '</select><br><br>'; //echo '<br>'; echo '</fieldset>'; echo '</div>'; } echo '<input type="submit" value="Submit form">'; echo '</form>'; here's checkout.php: if(isset($_POST['guest'])){ foreach($_POST['guest'] as $guest){ $guest['first_name'] = strip_tags($guest['first_name']); $guest['surname'] = strip_tags($guest['surname']); } //$_SESSION['guest'] = $guests; }

Read the article

Nested form problem in Rails : NoMethodError in Show

- by brianheys

I'm trying to build a simple product backlog application to teach myself Rails. For each product, there can be multiple product backlog entries, so I want to create a product view that shows the product information, all the backlog entries for the product, and includes a nested form for adding more backlog entries. Everything works until I try to add the form to the view, which then results in the following error: NoMethodError in Products#show Showing app/views/products/show.html.erb where line #29 raised: undefined method `pblog_ref' for #<Product:0x10423ba68> Extracted source (around line #29): 26: <%= f.error_messages %> 27: <p> 28: <%= f.label :pblog_ref %><br /> 29: <%= f.text_field :pblog_ref %> 30: </p> 31: <p> 32: <%= f.label :product %><br /> The product view where the problem is reported is as follows (the partial works fine, so I won't include that code): <h1>Showing product</h1> <p> <b>Product ref:</b> <%=h @product.product_ref %> </p> <p> <b>Description:</b> <%=h @product.description %> </p> <p> <b>Owner:</b> <%=h @product.owner %> </p> <p> <b>Status:</b> <%=h @product.status %> </p> <h2>Product backlog</h2> <div id="product-backlog"> <%= render :partial => @product.product_backlogs %> </div> <% form_for(@product, ProductBacklog.new) do |f| %> <%= f.error_messages %> <p> <%= f.label :pblog_ref %><br /> <%= f.text_field :pblog_ref %> </p> <p> <%= f.label :product %><br /> <%= f.text_field :product %> </p> <p> <%= f.label :description %><br /> <%= f.text_field :description %> </p> <p> <%= f.label :owner %><br /> <%= f.text_field :owner %> </p> <p> <%= f.label :status %><br /> <%= f.text_field :status %> </p> <p> <%= f.submit 'Create' %> </p> <% end %> <%= link_to 'Edit', edit_product_path(@product) %> | <%= link_to 'Back', products_path %> This is the Product model: class Product < ActiveRecord::Base validates_presence_of :product_ref, :description, :owner has_many :product_backlogs end This is the ProductBacklog model: class ProductBacklog < ActiveRecord::Base belongs_to :product end These are the routes: map.resources :product_backlogs map.resources :products, :has_many => :product_backlogs I've checked what I'm doing against the Creating a weblog in 15 minutes with Rails 2 screencast, and in principle I seem to be doing the same thing as him - only his nested comments form works, and mine doesn't! I hope someone can help with this, before I turn mad! I'm sure it's something trivial.

Read the article

c windows connect() fails. error 10049

- by Joshua Moore

The following two pieces of code compile, but I get a connect() failed error on the client side. (compiled with MinGW). Client Code: // thanks to cs.baylor.edu/~donahoo/practical/CSockets/code/TCPEchoClientWS.c #include <stdio.h> #include <winsock.h> #include <stdlib.h> #define RCVBUFSIZE 32 // size of receive buffer void DieWithError(char *errorMessage); int main(int argc, char* argv[]) { int sock; struct sockaddr_in echoServAddr; unsigned short echoServPort; char *servIP; char *echoString; char echoBuffer[RCVBUFSIZE]; int echoStringLen; int bytesRcvd, totalBytesRcvd; WSAData wsaData; if((argc < 3) || (argc > 4)){ fprintf(stderr, "Usage: %s <Sever IP> <Echo Word> [<Echo Port>]\n", argv[0]); exit(1); } if (argc==4) echoServPort = atoi(argv[3]); // use given port if any else echoServPort = 7; // echo is well-known port for echo service if(WSAStartup(MAKEWORD(2, 0), &wsaData) != 0){ // load winsock 2.0 dll fprintf(stderr, "WSAStartup() failed"); exit(1); } // create reliable, stream socket using tcp if((sock=socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0) DieWithError("socket() failed"); // construct the server address structure memset(&echoServAddr, 0, sizeof(echoServAddr)); echoServAddr.sin_family = AF_INET; echoServAddr.sin_addr.s_addr = inet_addr(servIP); // server IP address echoServAddr.sin_port = htons(echoServPort); // establish connection to the echo server if(connect(sock, (struct sockaddr*)&echoServAddr, sizeof(echoServAddr)) < 0) DieWithError("connect() failed"); echoStringLen = strlen(echoString); // determine input length // send the string, includeing the null terminator to the server if(send(sock, echoString, echoStringLen, 0)!= echoStringLen) DieWithError("send() sent a different number of bytes than expected"); totalBytesRcvd = 0; printf("Received: "); // setup to print the echoed string while(totalBytesRcvd < echoStringLen){ // receive up to the buffer size (minus 1 to leave space for a null terminator) bytes from the sender if(bytesRcvd = recv(sock, echoBuffer, RCVBUFSIZE-1, 0) <= 0) DieWithError("recv() failed or connection closed prematurely"); totalBytesRcvd += bytesRcvd; // keep tally of total bytes echoBuffer[bytesRcvd] = '\0'; printf("%s", echoBuffer); // print the echo buffer } printf("\n"); closesocket(sock); WSACleanup(); exit(0); } void DieWithError(char *errorMessage) { fprintf(stderr, "%s: %d\n", errorMessage, WSAGetLastError()); exit(1); } Server Code: // thanks cs.baylor.edu/~donahoo/practical/CSockets/code/TCPEchoServerWS.c #include <stdio.h> #include <winsock.h> #include <stdlib.h> #define MAXPENDING 5 // maximum outstanding connection requests #define RCVBUFSIZE 1000 void DieWithError(char *errorMessage); void HandleTCPClient(int clntSocket); // tcp client handling function int main(int argc, char **argv) { int serverSock; int clientSock; struct sockaddr_in echoServerAddr; struct sockaddr_in echoClientAddr; unsigned short echoServerPort; int clientLen; // length of client address data structure WSAData wsaData; if (argc!=2){ fprintf(stderr, "Usage: %s <Server Port>\n", argv[0]); exit(1); } echoServerPort = atoi(argv[1]); if(WSAStartup(MAKEWORD(2, 0), &wsaData)!=0){ fprintf(stderr, "WSAStartup() failed"); exit(1); } // create socket for incoming connections if((serverSock=socket(PF_INET, SOCK_STREAM, IPPROTO_TCP))<0) DieWithError("socket() failed"); // construct local address structure memset(&echoServerAddr, 0, sizeof(echoServerAddr)); echoServerAddr.sin_family = AF_INET; echoServerAddr.sin_addr.s_addr = htonl(INADDR_ANY); // any incoming interface echoServerAddr.sin_port = htons(echoServerPort); // local port // bind to the local address if(bind(serverSock, (struct sockaddr*)&echoServerAddr, sizeof(echoServerAddr) )<0) DieWithError("bind() failed"); // mark the socket so it will listen for incoming connections if(listen(serverSock, MAXPENDING)<0) DieWithError("listen() failed"); for (;;){ // run forever // set the size of the in-out parameter clientLen = sizeof(echoClientAddr); // wait for a client to connect if((clientSock = accept(serverSock, (struct sockaddr*)&echoClientAddr, &clientLen)) < 0) DieWithError("accept() failed"); // clientSock is connected to a client printf("Handling client %s\n", inet_ntoa(echoClientAddr.sin_addr)); HandleTCPClient(clientSock); } // NOT REACHED } void DieWithError(char *errorMessage) { fprintf(stderr, "%s: %d\n", errorMessage, WSAGetLastError()); exit(1); } void HandleTCPClient(int clientSocket) { char echoBuffer[RCVBUFSIZE]; // buffer for echostring int recvMsgSize; // size of received message // receive message from client if((recvMsgSize = recv(clientSocket, echoBuffer, RCVBUFSIZE, 0) <0)) DieWithError("recv() failed"); // send received string and receive again until end of transmission while(recvMsgSize > 0){ // echo message back to client if(send(clientSocket, echoBuffer, recvMsgSize, 0)!=recvMsgSize) DieWithError("send() failed"); // see if there's more data to receive if((recvMsgSize = recv(clientSocket, echoBuffer, RCVBUFSIZE, 0)) <0) DieWithError("recv() failed"); } closesocket(clientSocket); // close client socket } How can I fix this?

Read the article

Compile error with initializer_list when trying to use it to initialize member value of class

- by ilektron

I am trying to make a class initializable from an initialization_list in a class constructor's constructor's initialization list. It works for a std::map, but not for my custom class. I don't see any difference other than templates are used in std::map. #include <iostream> #include <initializer_list> #include <string> #include <sstream> #include <map> using std::string; class text_thing { private: string m_text; public: text_thing() { } text_thing(text_thing& other); text_thing(std::initializer_list< std::pair<const string, const string> >& il); text_thing& operator=(std::initializer_list< std::pair<const string, const string> >& il); operator string() { return m_text; } }; class static_base { private: std::map<string, string> m_test_map; text_thing m_thing; static_base(); public: static static_base& getInstance() { static static_base instance; return instance; } string getText() { return (string)m_thing; } }; typedef std::pair<const string, const string> spair; text_thing::text_thing(text_thing& other) { m_text = other.m_text; } text_thing::text_thing(std::initializer_list< std::pair<const string, const string> >& il) { std::stringstream text_gen; for (auto& apair : il) { text_gen << "{" << apair.first << ", " << apair.second << "}" << std::endl; } } text_thing& text_thing::operator=(std::initializer_list< std::pair<const string, const string> >& il) { std::stringstream text_gen; for (auto& apair : il) { text_gen << "{" << apair.first << ", " << apair.second << "}" << std::endl; } return *this; } static_base::static_base() : m_test_map{{"test", "1"}, {"test2", "2"}}, // Compiler fine with this m_thing{{"test", "1"}, {"test2", "2"}} // Compiler doesn't like this { } int main() { std::cout << "Starting the program" << std::endl; std::cout << "The text thing: " << std::endl << static_base::getInstance().getText(); } I get this compiler output g++ -O0 -g3 -Wall -c -fmessage-length=0 -std=c++11 -MMD -MP -MF"static_base.d" -MT"static_base.d" -o "static_base.o" "../static_base.cpp" Finished building: ../static_base.cpp Building file: ../test.cpp Invoking: GCC C++ Compiler g++ -O0 -g3 -Wall -c -fmessage-length=0 -std=c++11 -MMD -MP -MF"test.d" -MT"test.d" -o "test.o" "../test.cpp" ../test.cpp: In constructor ‘static_base::static_base()’: ../test.cpp:94:40: error: no matching function for call to ‘text_thing::text_thing(<brace-enclosed initializer list>)’ m_thing{{"test", "1"}, {"test2", "2"}} ^ ../test.cpp:94:40: note: candidates are: ../test.cpp:72:1: note: text_thing::text_thing(std::initializer_list<std::pair<const std::basic_string<char>, const std::basic_string<char> > >&) text_thing::text_thing(std::initializer_list< std::pair<const string, const string> >& il) ^ ../test.cpp:72:1: note: candidate expects 1 argument, 2 provided ../test.cpp:67:1: note: text_thing::text_thing(text_thing&) text_thing::text_thing(text_thing& other) ^ ../test.cpp:67:1: note: candidate expects 1 argument, 2 provided ../test.cpp:23:2: note: text_thing::text_thing() text_thing() ^ ../test.cpp:23:2: note: candidate expects 0 arguments, 2 provided make: *** [test.o] Error 1 Output of gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.1-2ubuntu1~13.04' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.8.1 (Ubuntu 4.8.1-2ubuntu1~13.04) It compiles fine with the std::map constructed this way, and if I modify the static_base to return the strings from the maps, all is fine and dandy. Please help me understand what is going on here.

Read the article

Swing object: first setText() gets "stuck" when using Mac Java SE 6

- by Tim

Hi there, I am a Java newbie trying to maintain an application that works fine under J2SE 5.0 (32- and 64-bit) but has a very specific problem when run under Java SE 6 64-bit: [Tims-MPB:~] tlynch% java -version java version "1.6.0_15" Java(TM) SE Runtime Environment (build 1.6.0_15-b03-226) Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-92, mixed mode) The application is cross-platform and reportedly works correctly on Java SE 6 under Windows, though I haven't been able to verify that myself. The program uses a JTextField for some text entry and a JLabel to indicate the text to be entered. The first time the showDialog() method is called to set the label text and display the dialog, it works correctly, but subsequent calls all result in the display of the label from the initial invocation rather than the one most recently specified via setText(). public void showDialog(String msgText) { System.out.println("set ChatDialog: " + msgText); jLabel1.setText(msgText); jLabel1.repaint(); // I added this; it didn't help System.out.println("get ChatDialog: " + jLabel1.getText()); super.setVisible(true); } [the full text of the class is provided below] The added printlns validate that expected text is passed to the label's setText() method and is confirmed by retrieving it using getText(), but what shows up on the screen/GUI is always the text from the very first time the method was called for the object. A similar issue is observed with a JTextArea used to label another dialog box. These problem are consistent across multiple Mac systems running Java SE 6 under OS 10.5.x and 10.6.x, but they are never observed when one reverts to J2SE 5.0. If there is some background information pertinent to this problem that I have omitted, please let me know. Any insights or advice appreciated. package gui; import java.awt.*; import java.awt.event.KeyEvent; import javax.swing.*; // Referenced classes of package gui: // MyJPanel, ChatDialog_jTextField1_keyAdapter, WarWindow public class ChatDialog extends JDialog { public ChatDialog(JFrame parent, WarWindow w) { super(parent, true); text = ""; borderLayout1 = new BorderLayout(); jPanel1 = new MyJPanel(); borderLayout2 = new BorderLayout(); jPanel2 = new MyJPanel(); jPanel3 = new MyJPanel(); jLabel1 = new JLabel(); jTextField1 = new JTextField(); warWindow = w; try { jbInit(); } catch(Exception exception) { System.out.println("Problem with ChatDialog init"); exception.printStackTrace(); } return; } public String getText() { return text; } void jTextField1_keyPressed(KeyEvent e) { int id = e.getKeyCode(); switch(id) { case 10: // '\n' text = jTextField1.getText(); setVisible(false); break; } } private void jbInit() throws Exception { setLocation(232, 450); setSize(560, 60); setModal(true); setResizable(false); setUndecorated(true); getContentPane().setLayout(borderLayout1); jPanel1.setLayout(borderLayout2); jPanel2.setMinimumSize(new Dimension(10, 20)); jPanel2.setPreferredSize(new Dimension(10, 20)); jLabel1.setPreferredSize(new Dimension(380, 15)); jLabel1.setHorizontalAlignment(0); jLabel1.setText("Chat Message"); jTextField1.setPreferredSize(new Dimension(520, 21)); jTextField1.setRequestFocusEnabled(false); jTextField1.addKeyListener(new ChatDialog_jTextField1_keyAdapter(this)); getContentPane().add(jPanel1, "Center"); jPanel1.add(jPanel2, "North"); jPanel2.add(jLabel1, null); jPanel1.add(jPanel3, "Center"); jPanel3.add(jTextField1, null); } public void setVisible(boolean b) { jTextField1.setText(""); super.setVisible(b); } public void showDialog(String msgText) { System.out.println("set ChatDialog: " + msgText); jLabel1.setText(msgText); jLabel1.repaint(); // I added this; it didn't help System.out.println("get ChatDialog: " + jLabel1.getText()); super.setVisible(true); } void this_keyPressed(KeyEvent e) { int id = e.getKeyCode(); switch(id) { case 10: // '\n' System.exit(88); break; } } BorderLayout borderLayout1; BorderLayout borderLayout2; JLabel jLabel1; JPanel jPanel1; JPanel jPanel2; JPanel jPanel3; JTextField jTextField1; String text; WarWindow warWindow; }

Read the article

WCF Service error received when using TCP: "The message could not be dispatched..."

- by StM

I am new to creating WCF services. I have created a WCF web service in VS2008 that is running on IIS 7. When I use http the service works perfectly. When I configure the service for TCP and run I get the following error message. There was a communication problem. The message could not be dispatched because the service at the endpoint address 'net:tcp://elec:9090/CoordinateIdTool_Tcp/IdToolService.svc is unavailable for the protocol of the address. I have searched a lot of forums, including this one, for a resolution but nothing has worked. Everything appears to be set up correctly on IIS 7. WAS has been set up to run. The default web site has a net.tcp binding and the application has net.tcp under the enabled protocols. I am including what I think is the important part of the web.config from the host project and also the app.config from the client project I am using to test the service. Hopefully someone can spot my error. Thanks in advance for any help or recommendations that anyone can provide. Web.Config <bindings> <wsHttpBinding> <binding name="wsHttpBindingNoMsgs"> <security mode="None" /> </binding> </wsHttpBinding> </bindings> <services> <service behaviorConfiguration="CogIDServiceHost.ServiceBehavior" name="CogIDServiceLibrary.CogIdService"> <endpoint address="" binding="wsHttpBinding" bindingConfiguration="wsHttpBindingNoMsgs" contract="CogIDServiceLibrary.CogIdTool"> <identity> <dns value="localhost" /> </identity> </endpoint> <endpoint address="mex" binding="mexHttpBinding" bindingConfiguration="" contract="IMetadataExchange" /> <endpoint name="CoordinateIdService_TCP" address="net.tcp://elec:9090/CoordinateIdTool_Tcp/IdToolService.svc" binding="netTcpBinding" bindingConfiguration="" contract="CogIDServiceLibrary.CogIdTool"> <identity> <dns value="localhost" /> </identity> </endpoint> </service> </services> <behaviors> <serviceBehaviors> <behavior name="CogIDServiceHost.ServiceBehavior"> <serviceMetadata httpGetEnabled="true" /> <serviceDebug includeExceptionDetailInFaults="false" /> </behavior> </serviceBehaviors> </behaviors> App.Config <system.serviceModel> <diagnostics performanceCounters="Off"> <messageLogging logEntireMessage="true" logMalformedMessages="false" logMessagesAtServiceLevel="false" logMessagesAtTransportLevel="false" /> </diagnostics> <behaviors /> <bindings> <wsHttpBinding> <binding name="WSHttpBinding_CogIdTool" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false"> <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" /> <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" /> <security mode="None"> <transport clientCredentialType="Windows" proxyCredentialType="None" realm="" /> <message clientCredentialType="Windows" negotiateServiceCredential="true" establishSecurityContext="true" /> </security> </binding> <binding name="wsHttpBindingNoMsg"> <security mode="None"> <transport clientCredentialType="Windows" /> <message clientCredentialType="Windows" /> </security> </binding> </wsHttpBinding> </bindings> <client> <endpoint address="http://sdet/CogId_WCF/IdToolService.svc" binding="wsHttpBinding" bindingConfiguration="wsHttpBindingNoMsg" contract="CogIdServiceReference.CogIdTool" name="IISHostWsHttpBinding"> <identity> <dns value="localhost" /> </identity> </endpoint> <endpoint address="http://localhost:1890/IdToolService.svc" binding="wsHttpBinding" bindingConfiguration="WSHttpBinding_CogIdTool" contract="CogIdServiceReference.CogIdTool" name="WSHttpBinding_CogIdTool"> <identity> <dns value="localhost" /> </identity> </endpoint> <endpoint address="http://elec/CoordinateIdTool/IdToolService.svc" binding="wsHttpBinding" bindingConfiguration="wsHttpBindingNoMsg" contract="CogIdServiceReference.CogIdTool" name="IIS7HostWsHttpBinding_Elec"> <identity> <dns value="localhost" /> </identity> </endpoint> <endpoint address="net.tcp://elec:9090/CoordinateIdTool_Tcp/IdToolService.svc" binding="netTcpBinding" bindingConfiguration="" contract="CogIdServiceReference.CogIdTool" name="IIS7HostTcpBinding_Elec" > <identity> <dns value="localhost"/> </identity> </endpoint> </client> </system.serviceModel>

Read the article

SWIG & C/C++ Python API connected - SEGFAULT

- by user289637

Hello, my task is to create dual program. At the beginning I start C program that calls throught C/C++ API of Python some Python method. The called method after that call a function that is created with SWIG. I show you my sample also with backtrace from gdb after I am given Segmentation fault. main.c: #include <Python.h> #include <stdio.h> #include "utils.h" int main(int argc, char** argv) { printf("Calling from C !\n"); increment(); int i; for(i = 0; i < 11; ++i) { Py_Initialize(); PyObject *pname = PyString_FromString("py_function"); PyObject *module = PyImport_Import(pname); PyObject *dict = PyModule_GetDict(module); PyObject *func = PyDict_GetItemString(dict, "ink"); PyObject_CallObject(func, NULL); Py_DECREF(module); Py_DECREF(pname); printf("\tbefore finalize\n"); Py_Finalize(); printf("\tafter finalize\n"); } return 0; } utils.c #include <stdio.h> #include "utils.h" void increment(void) { printf("Incremention counter to: %u\n", ++counter); } py_function.py #!/usr/bin/python2.6 '''py_function.py - Python source designed to demonstrate the use of python embedding''' import utils def ink(): print 'I am gonna increment !' utils.increment() and last think is my Makefile & SWIG configure file Makefile: CC=gcc CFLAGS=-c -g -Wall -std=c99 all: main main: main.o utils.o utils_wrap.o $(CC) main.o utils.o -lpython2.6 -o sample swig -Wall -python -o utils_wrap.c utils.i $(CC) utils.o utils_wrap.o -shared -o _utils.so main.o: main.c $(CC) $(CFLAGS) main.c -I/usr/include/python2.6 -o main.o utils.o: utils.c utils.h $(CC) $(CFLAGS) -fPIC utils.c -o $@ utils_wrap.o: utils_wrap.c $(CC) -c -fPIC utils_wrap.c -I/usr/include/python2.6 -o $@ clean: rm -rf *.o The program is called by ./main and there is output: (gdb) run Starting program: /home/marxin/Programming/python2/sample [Thread debugging using libthread_db enabled] Calling from C ! Incremention counter to: 1 I am gonna increment ! Incremention counter to: 2 before finalize after finalize I am gonna increment ! Incremention counter to: 3 before finalize after finalize I am gonna increment ! Incremention counter to: 4 before finalize after finalize Program received signal SIGSEGV, Segmentation fault. 0xb7ed3e4e in PyObject_Malloc () from /usr/lib/libpython2.6.so.1.0 Backtrace: (gdb) backtrace #0 0xb7ed3e4e in PyObject_Malloc () from /usr/lib/libpython2.6.so.1.0 #1 0xb7ca2b2c in ?? () #2 0xb7f8dd40 in ?? () from /usr/lib/libpython2.6.so.1.0 #3 0xb7eb014c in ?? () from /usr/lib/libpython2.6.so.1.0 #4 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #5 0xb7f99820 in ?? () from /usr/lib/libpython2.6.so.1.0 #6 0x00000001 in ?? () #7 0xb7f8dd40 in ?? () from /usr/lib/libpython2.6.so.1.0 #8 0xb7f4f014 in _PyObject_GC_Malloc () from /usr/lib/libpython2.6.so.1.0 #9 0xb7f99820 in ?? () from /usr/lib/libpython2.6.so.1.0 #10 0xb7f4f104 in _PyObject_GC_NewVar () from /usr/lib/libpython2.6.so.1.0 #11 0xb7ee8760 in _PyType_Lookup () from /usr/lib/libpython2.6.so.1.0 #12 0xb7f99820 in ?? () from /usr/lib/libpython2.6.so.1.0 #13 0x00000001 in ?? () #14 0xb7f8dd40 in ?? () from /usr/lib/libpython2.6.so.1.0 #15 0xb7ef13ed in ?? () from /usr/lib/libpython2.6.so.1.0 #16 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #17 0x00000001 in ?? () #18 0xbfff0c34 in ?? () #19 0xb7e993c3 in ?? () from /usr/lib/libpython2.6.so.1.0 #20 0x00000001 in ?? () #21 0xbfff0c70 in ?? () #22 0xb7f99da0 in ?? () from /usr/lib/libpython2.6.so.1.0 #23 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #24 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #25 0x080a6b0c in ?? () #26 0x080a6b0c in ?? () #27 0xb7e99420 in PyObject_CallFunctionObjArgs () from /usr/lib/libpython2.6.so.1.0 #28 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #29 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #30 0x800e55eb in ?? () #31 0x080a6b0c in ?? () #32 0xb7e9958c in PyObject_IsSubclass () from /usr/lib/libpython2.6.so.1.0 #33 0xb7f8dd40 in ?? () from /usr/lib/libpython2.6.so.1.0 #34 0x080a9020 in ?? () #35 0xb7fb78f0 in PyFPE_counter () from /usr/lib/libpython2.6.so.1.0 #36 0xb7f86ff4 in ?? () from /usr/lib/libpython2.6.so.1.0 #37 0x00000000 in ?? () Thanks for your help and advices, marxin

Read the article

Mysql table data problem?

- by DaTeNtImE

I'm new to mysql and was wondering how can I add the users birthdate in the following HTML format to the MYSQL table data listed below? How would the structure look like for example email VARCHAR(80) NOT NULL,? Here is the HTML code below. <li><label>Date of Birth: </label> <label for="month">Month: </label> <select name="month" id="month"> <option value="January">January</option> <option value="February">February</option> <option value="March">March</option> <option value="April">April</option> <option value="May">May</option> <option value="June">June</option> <option value="July">July</option> <option value="August">August</option> <option value="September">September</option> <option value="October">October</option> <option value="November">November</option> <option value="December">December</option> </select> <label for="day">Day: </label> <select id="day" name="day"> <option value="0" selected="selected">Day</option> <option value="1">1</option> <option value="2">2</option> <option value="3">3</option> <option value="4">4</option> <option value="5">5</option> <option value="6">6</option> <option value="7">7</option> <option value="8">8</option> <option value="9">9</option> <option value="10">10</option> <option value="11">11</option> <option value="12">12</option> <option value="13">13</option> <option value="14">14</option> <option value="15">15</option> <option value="16">16</option> <option value="17">17</option> <option value="18">18</option> <option value="19">19</option> <option value="20">20</option> <option value="21">21</option> <option value="22">22</option> <option value="23">23</option> <option value="24">24</option> <option value="25">25</option> <option value="26">26</option> <option value="27">27</option> <option value="28">28</option> <option value="29">29</option> <option value="30">30</option> <option value="31">31</option> </select> <label for="year">Year: </label><input type="text" name="year" id="year" /></li> Here is the MySQL table data. CREATE TABLE users ( user_id INT UNSIGNED NOT NULL AUTO_INCREMENT, first_name VARCHAR(20) NOT NULL, last_name VARCHAR(40) NOT NULL, email VARCHAR(80) NOT NULL, pass CHAR(40) NOT NULL, user_level TINYINT(1) UNSIGNED NOT NULL DEFAULT 0, active CHAR(32), registration_date DATETIME NOT NULL, PRIMARY KEY (user_id), UNIQUE KEY (email), INDEX login (email, pass) );

Read the article

How to create an attached-property to change a resource's property?

- by king.net

I have a DrawingBrush as a resource like this: <DrawingBrush x:Key="Calendar" Stretch="Uniform"> <DrawingBrush.Drawing> <DrawingGroup> <DrawingGroup.Children> <GeometryDrawing Geometry="F1 M 28.0917,2.13333C 42.4005,2.13333 54,13.7329 54,28.0417C 54,42.3504 42.4004,53.95 28.0917,53.95C 13.7829,53.95 2.18334,42.3504 2.18334,28.0417C 2.18334,13.7329 13.7829,2.13333 28.0917,2.13333 Z "> <GeometryDrawing.Pen> <Pen Thickness="4" LineJoin="Round" Brush="#FF000000"/> </GeometryDrawing.Pen> </GeometryDrawing> <GeometryDrawing Geometry="F1 M 16.9667,16.7083L 39.7167,16.7083L 39.7167,41.625L 16.9667,41.625L 16.9667,16.7083 Z "> <GeometryDrawing.Pen> <Pen Thickness="2.66667" StartLineCap="Square" EndLineCap="Square" MiterLimit="2.75" Brush="#FF000000"/> </GeometryDrawing.Pen> </GeometryDrawing> <GeometryDrawing Brush="#FF000000" Geometry="F1 M 15.6333,15.9583L 40.7167,15.9583L 40.7167,25.2917L 15.6333,25.2917L 15.6333,15.9583 Z "/> <GeometryDrawing Brush="#FFFFFFFF" Geometry="F1 M 18.2167,11.9583L 22.9667,11.9583L 22.9667,20.875L 18.2167,20.875L 18.2167,11.9583 Z "> <GeometryDrawing.Pen> <Pen Thickness="1.33333" StartLineCap="Square" EndLineCap="Square" MiterLimit="2.75" Brush="#FF000000"/> </GeometryDrawing.Pen> </GeometryDrawing> <GeometryDrawing Brush="#FFFFFFFF" Geometry="F1 M 33.7167,11.925L 38.4667,11.925L 38.4667,20.8417L 33.7167,20.8417L 33.7167,11.925 Z "> <GeometryDrawing.Pen> <Pen Thickness="1.33333" StartLineCap="Square" EndLineCap="Square" MiterLimit="2.75" Brush="#FF000000"/> </GeometryDrawing.Pen> </GeometryDrawing> <GeometryDrawing Brush="#FF000000" Geometry="F1 M 28.0154,36.2658L 28.0154,37.4894L 21.6254,37.4894C 21.6169,37.1934 21.6615,36.908 21.7592,36.6333C 21.915,36.1815 22.165,35.7425 22.5091,35.3162C 22.8533,34.8899 23.3617,34.4091 24.0344,33.8738C 25.0782,32.983 25.776,32.295 26.1279,31.81C 26.4799,31.3249 26.6558,30.8551 26.6558,30.4005C 26.6558,29.9473 26.4894,29.5653 26.1566,29.2544C 25.8238,28.9435 25.3904,28.7881 24.8565,28.7881C 24.2915,28.7881 23.8393,28.9442 23.5001,29.2565C 23.161,29.5688 22.9892,30.0018 22.985,30.5556L 21.7614,30.4196C 21.8449,29.5345 22.1576,28.86 22.6993,28.3962C 23.241,27.9323 23.9686,27.7004 24.882,27.7004C 25.8054,27.7004 26.5358,27.9596 27.0733,28.4779C 27.6107,28.9963 27.8795,29.6385 27.8795,30.4047C 27.8795,30.7942 27.8065,31.1769 27.6607,31.5529C 27.5148,31.9289 27.2726,32.3251 26.9341,32.7415C 26.5957,33.1579 26.0115,33.7215 25.1816,34.4325C 24.4692,35.0216 24.0008,35.4214 23.7763,35.6317C 23.5518,35.842 23.3667,36.0533 23.2208,36.2658L 28.0154,36.2658 Z "/> <GeometryDrawing Brush="#FF000000" Geometry="F1 M 33.3178,37.4894L 33.3178,35.1781L 28.9671,35.1781L 28.9671,33.9545L 33.5897,27.8364L 34.5414,27.8364L 34.5414,33.9545L 35.765,33.9545L 35.765,35.1781L 34.5414,35.1781L 34.5414,37.4894L 33.3178,37.4894 Z M 33.3178,33.9545L 33.3178,30.1774L 30.4648,33.9545L 33.3178,33.9545 Z "/> </DrawingGroup.Children> </DrawingGroup> </DrawingBrush.Drawing> </DrawingBrush> And I can use it like this: <Rectangle Fill="{DynamicResource Calender}" /> Now, my question is: how can I create an attached-property to change all brushes on my resource? e.g. I be able to create this: <Rectangle Fill="{DynamicResource Calendar}" attached:IconHelper.Foreground="Blue" /> on my Rectangle and in my resource, I can get: <DrawingBrush x:Key="Calendar" Stretch="Uniform"> <DrawingBrush.Drawing> <DrawingGroup> <DrawingGroup.Children> <GeometryDrawing Geometry="blah blah"> <GeometryDrawing.Pen> <Pen Brush={attached:ReadItFromAboveRectangle}/> </GeometryDrawing.Pen> </GeometryDrawing> <GeometryDrawing Geometry="blah blah"> <GeometryDrawing.Pen> <Pen Brush={attached:ReadItFromAboveRectangle}/> </GeometryDrawing.Pen>  Is there any way to read an attached-property on Rectangle in Calendar resource? Or is there any other way to do this? Thanks in advance.

Read the article

Django + dbxml + Apache = problems. Any solutions?

- by Jason

I'm trying to set up a Django application using WSGI. That works fine. However, I am having some issues with part of my Django app that uses BDB XML. My Apache config is as follows: Listen 8000 WSGISocketPrefix /tmp/wsgi <VirtualHost *:8000> ServerName <server name> DocumentRoot <path to doc root> LogLevel info WSGIScriptAlias / <path to wsgi> WSGIApplicationGroup %{GLOBAL} WSGIDaemonProcess debug threads=1 WSGIProcessGroup debug </VirtualHost> However, I'm still getting the following error: DB_ENV->repmgr_stat interface requires an environment configured for the replication subsystem [error] child died with signal 11 My environment is opened as: environment = DBEnv() environment.open( <absolute db env path>, DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL, 0 ) I am using: python 2.6.2 apache 2.2 ubuntu 9.04 dbxml 2.5.13 compiled from source (so libdb-4.8, bsddb3, all that jazz) I see Apache seems to link to libdb-4.6. Is this a problem? ldd /usr/sbin/apache2 | grep libdb libdb-4.6.so => /usr/lib/libdb-4.6.so (0xb7c01000) Updated Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb5a48b90 (LWP 12700)] 0x00000000 in ?? () (gdb) thread apply all bt Thread 4 (Thread 0xb6a67b90 (LWP 12698)): #0 0xb7f11422 in __kernel_vsyscall () #1 0xb7de07b1 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7ea5bcf in apr_sleep () from /usr/lib/libapr-1.so.0 #3 0xb6d7afee in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #4 0xb7ea38ec in ?? () from /usr/lib/libapr-1.so.0 #5 0xb7e6d4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7de849e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 3 (Thread 0xb6249b90 (LWP 12699)): #0 0xb7f11422 in __kernel_vsyscall () #1 0xb7de07b1 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7ea5bcf in apr_sleep () from /usr/lib/libapr-1.so.0 #3 0xb6d7ab39 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #4 0xb7ea38ec in ?? () from /usr/lib/libapr-1.so.0 #5 0xb7e6d4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7de849e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 2 (Thread 0xb5a48b90 (LWP 12700)): #0 0x00000000 in ?? () #1 0xb4f03b5e in DbXml::XmlManager::XmlManager () from /home/jason/dbxml-2.5.13/install/lib/libdbxml-2.5.so #2 0xb501b29b in _wrap_new_XmlManager (self=0x0, args=0xac66fcc) at dbxml_python_wrap.cpp:5183 #3 0xb6b77aed in PyCFunction_Call () from /usr/lib/libpython2.6.so.1.0 #4 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #5 0xb6bd70b5 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #6 0xb6bdb910 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so.1.0 #7 0xb6b6187a in ?? () from /usr/lib/libpython2.6.so.1.0 #8 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #9 0xb6b427a8 in ?? () from /usr/lib/libpython2.6.so.1.0 #10 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #11 0xb6b9ae03 in ?? () from /usr/lib/libpython2.6.so.1.0 #12 0xb6b90f55 in ?? () from /usr/lib/libpython2.6.so.1.0 #13 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #14 0xb6bd7618 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #15 0xb6bdb910 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so.1.0 #16 0xb6b6187a in ?? () from /usr/lib/libpython2.6.so.1.0 #17 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #18 0xb6b427a8 in ?? () from /usr/lib/libpython2.6.so.1.0 #19 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #20 0xb6bd3a34 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.6.so.1.0 #21 0xb6b44a7d in PyInstance_New () from /usr/lib/libpython2.6.so.1.0 #22 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #23 0xb6bd7618 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #24 0xb6bdb910 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so.1.0 #25 0xb6b61969 in ?? () from /usr/lib/libpython2.6.so.1.0 #26 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #27 0xb6bd70b5 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #28 0xb6bdb910 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so.1.0 #29 0xb6b61969 in ?? () from /usr/lib/libpython2.6.so.1.0 #30 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #31 0xb6b427a8 in ?? () from /usr/lib/libpython2.6.so.1.0 #32 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #33 0xb6b9b483 in ?? () from /usr/lib/libpython2.6.so.1.0 #34 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #35 0xb6bd70b5 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #36 0xb6bdab4f in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #37 0xb6bdb910 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so.1.0 #38 0xb6b6187a in ?? () from /usr/lib/libpython2.6.so.1.0 #39 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #40 0xb6b427a8 in ?? () from /usr/lib/libpython2.6.so.1.0 #41 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #42 0xb6b9b483 in ?? () from /usr/lib/libpython2.6.so.1.0 #43 0xb6b3198c in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #44 0xb6bd3a34 in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.6.so.1.0 #45 0xb6d7172d in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #46 0xb6d7539f in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #47 0xb6d7e1d8 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #48 0xb6d7a42c in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #49 0xb6d7a8bd in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #50 0xb6d7a9c5 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #51 0xb7ea38ec in ?? () from /usr/lib/libapr-1.so.0 #52 0xb7e6d4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #53 0xb7de849e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 1 (Thread 0xb7460b00 (LWP 12697)): #0 0xb7f11422 in __kernel_vsyscall () #1 0xb7e75300 in sigwait () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7ea3f3b in apr_signal_thread () from /usr/lib/libapr-1.so.0 #3 0xb6d7b48d in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #4 0xb6d7bc98 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #5 0xb6d79632 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so #6 0xb7e9a2c9 in apr_proc_other_child_alert () from /usr/lib/libapr-1.so.0 #7 0x08092202 in ap_mpm_run () #8 0x080673c8 in main () #0 0x00000000 in ?? ()

Read the article

OpenGL texture shifted somewhat to the left when applied to a quad

- by user308226

I'm a bit new to OpenGL and I've been having a problem with using textures. The texture seems to load fine, but when I run the program, the texture displays shifted a couple pixels to the left, with the section cut off by the shift appearing on the right side. I don't know if the problem here is in the my TGA loader or if it's the way I'm applying the texture to the quad. Here is the loader: #include "texture.h" #include <iostream> GLubyte uncompressedheader[12] = {0,0, 2,0,0,0,0,0,0,0,0,0}; GLubyte compressedheader[12] = {0,0,10,0,0,0,0,0,0,0,0,0}; TGA::TGA() { } //Private loading function called by LoadTGA. Loads uncompressed TGA files //Returns: TRUE on success, FALSE on failure bool TGA::LoadCompressedTGA(char *filename,ifstream &texturestream) { return false; } bool TGA::LoadUncompressedTGA(char *filename,ifstream &texturestream) { cout << "G position status:" << texturestream.tellg() << endl; texturestream.read((char*)header, sizeof(header)); //read 6 bytes into the file to get the tga header width = (GLuint)header[1] * 256 + (GLuint)header[0]; //read and calculate width and save height = (GLuint)header[3] * 256 + (GLuint)header[2]; //read and calculate height and save bpp = (GLuint)header[4]; //read bpp and save cout << bpp << endl; if((width <= 0) || (height <= 0) || ((bpp != 24) && (bpp !=32))) //check to make sure the height, width, and bpp are valid { return false; } if(bpp == 24) { type = GL_RGB; } else { type = GL_RGBA; } imagesize = ((bpp/8) * width * height); //determine size in bytes of the image cout << imagesize << endl; imagedata = new GLubyte[imagesize]; //allocate memory for our imagedata variable texturestream.read((char*)imagedata,imagesize); //read according the the size of the image and save into imagedata for(GLuint cswap = 0; cswap < (GLuint)imagesize; cswap += (bpp/8)) //loop through and reverse the tga's BGR format to RGB { imagedata[cswap] ^= imagedata[cswap+2] ^= //1st Byte XOR 3rd Byte XOR 1st Byte XOR 3rd Byte imagedata[cswap] ^= imagedata[cswap+2]; } texturestream.close(); //close ifstream because we're done with it cout << "image loaded" << endl; glGenTextures(1, &texID); // Generate OpenGL texture IDs glBindTexture(GL_TEXTURE_2D, texID); // Bind Our Texture glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); // Linear Filtered glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); glTexImage2D(GL_TEXTURE_2D, 0, type, width, height, 0, type, GL_UNSIGNED_BYTE, imagedata); delete imagedata; return true; } //Public loading function for TGA images. Opens TGA file and determines //its type, if any, then loads it and calls the appropriate function. //Returns: TRUE on success, FALSE on failure bool TGA::loadTGA(char *filename) { cout << width << endl; ifstream texturestream; texturestream.open(filename,ios::binary); texturestream.read((char*)header,sizeof(header)); //read 6 bytes into the file, its the header. //if it matches the uncompressed header's first 6 bytes, load it as uncompressed LoadUncompressedTGA(filename,texturestream); return true; } GLubyte* TGA::getImageData() { return imagedata; } GLuint& TGA::getTexID() { return texID; } And here's the quad: void Square::show() { glEnable(GL_TEXTURE_2D); glBindTexture(GL_TEXTURE_2D, texture.texID); //Move to offset glTranslatef( x, y, 0 ); //Start quad glBegin( GL_QUADS ); //Set color to white glColor4f( 1.0, 1.0, 1.0, 1.0 ); //Draw square glTexCoord2f(0.0f, 0.0f); glVertex3f( 0, 0, 0 ); glTexCoord2f(1.0f, 0.0f); glVertex3f( SQUARE_WIDTH, 0, 0 ); glTexCoord2f(1.0f, 1.0f); glVertex3f( SQUARE_WIDTH, SQUARE_HEIGHT, 0 ); glTexCoord2f(0.0f, 1.0f); glVertex3f( 0, SQUARE_HEIGHT, 0 ); //End quad glEnd(); //Reset glLoadIdentity(); }

Read the article

How to? WCF customBinding over Https

- by user663414

Hi all, I'm trying to setup a WCF service for internal use, on our external facing web-farm (we dont have a web farm internally, and I need this service to have failover and load-balancing). Requirements: PerSession state, as we need the service to retain variable data for each session. HTTPS. After lots of googling i've read I needed to create a customBinding, which I've done, but not sure if it is correct. Larger message size, as one of the parameters is a byte[] array, which can be a max of 5mb. no requirement to manually edit the client-side app.config. ie, I need the Developer to just add the service reference, and then starts using the object without fiddly changing of app.config. Note: I've previously had this service working under HTTP correctly (using wsHttpBinding). I've also had it working under HTTPS, but it didn't support PerSession state, and lost internal variable values each function call. I'm currently getting this error from the test harness: Could not find default endpoint element that references contract 'AppMonitor.IAppMonitorWcfService' in the ServiceModel client configuration section. This might be because no configuration file was found for your application, or because no endpoint element matching this contract could be found in the client element. NOTE: The error is arising on an Test Harness EXE, that has the WCF service referenced directly under Service References. This is not the problem of an exe referencing another object, that then references the WCF service, that i've read about. The WSDL is showing correctly when browsing to the URL. Web.Config: <system.serviceModel> <services> <service name="AppMonitor.AppMonitorWcfService" behaviorConfiguration="ServiceBehavior"> <endpoint address="" binding="customBinding" bindingConfiguration="EnablePerSessionUnderHttps" contract="AppMonitor.IAppMonitorWcfService"/> <endpoint address="mex" binding="mexHttpsBinding" contract="IMetadataExchange" /> </service> </services> <bindings> <customBinding> <binding name="EnablePerSessionUnderHttps" maxReceivedMessageSize="5242880"> <reliableSession ordered="true"/> <textMessageEncoding> <readerQuotas maxDepth="64" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="4096" maxNameTableCharCount="16384" /> </textMessageEncoding> <httpsTransport authenticationScheme="Anonymous" requireClientCertificate="false"/> </binding> </customBinding> </bindings> <behaviors> <serviceBehaviors> <behavior name="ServiceBehavior"> <serviceMetadata httpsGetEnabled="true" httpGetEnabled="false"/> <serviceDebug includeExceptionDetailInFaults="true"/> </behavior> </serviceBehaviors> </behaviors> </system.serviceModel> EXE's App.config (auto-generated when adding the Service Reference): <configuration> <system.serviceModel> <bindings> <wsHttpBinding> <binding name="CustomBinding_IAppMonitorWcfService" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false"> <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" /> <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="true" /> <security mode="Transport"> <transport clientCredentialType="None" proxyCredentialType="None" realm="" /> <message clientCredentialType="Windows" negotiateServiceCredential="true" establishSecurityContext="true" /> </security> </binding> </wsHttpBinding> </bindings> <client /> </system.serviceModel> </configuration> I'm not sure why the app.config is showing wsHttpBinding? Shouldn't this be customBinding? I really dont want to have to edit the app.config, as this service will be used by dozens of developers, and I want them to just be able to add the Service Reference, and away they go... Using VS2008, .NET 3.51. I think server is IIS7, Win Server 2008, can confirm if needed.

Read the article

Sort multidimension array in php by more than two fields

- by Ankita Agrawal

Read the article

Trying to draw textured triangles on device fails, but the emulator works. Why?

- by Dinedal

I have a series of OpenGL-ES calls that properly render a triangle and texture it with alpha blending on the emulator (2.0.1). When I fire up the same code on an actual device (Droid 2.0.1), all I get are white squares. This suggests to me that the textures aren't loading, but I can't figure out why they aren't loading. All of my textures are 32-bit PNGs with alpha channels, under res/raw so they aren't optimized per the sdk docs. Here's how I am loading my textures: private void loadGLTexture(GL10 gl, Context context, int reasource_id, int texture_id) { //Get the texture from the Android resource directory Bitmap bitmap = BitmapFactory.decodeResource(context.getResources(), reasource_id, sBitmapOptions); //Generate one texture pointer... gl.glGenTextures(1, textures, texture_id); //...and bind it to our array gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[texture_id]); //Create Nearest Filtered Texture gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_NEAREST); gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR); //Different possible texture parameters, e.g. GL10.GL_CLAMP_TO_EDGE gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S, GL10.GL_REPEAT); gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT); //Use the Android GLUtils to specify a two-dimensional texture image from our bitmap GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bitmap, 0); //Clean up bitmap.recycle(); } Here's how I am rendering the texture: //Clear gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT); //Enable vertex buffer gl.glEnableClientState(GL10.GL_VERTEX_ARRAY); gl.glEnableClientState(GL10.GL_TEXTURE_COORD_ARRAY); gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBuffer); gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer); //Push transformation matrix gl.glPushMatrix(); //Transformation matrices gl.glTranslatef(x, y, 0.0f); gl.glScalef(scalefactor, scalefactor, 0.0f); gl.glColor4f(1.0f,1.0f,1.0f,1.0f); //Bind the texture gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[textureid]); //Draw the vertices as triangles gl.glDrawElements(GL10.GL_TRIANGLES, indices.length, GL10.GL_UNSIGNED_BYTE, indexBuffer); //Pop the matrix back to where we left it gl.glPopMatrix(); //Disable the client state before leaving gl.glDisableClientState(GL10.GL_VERTEX_ARRAY); gl.glDisableClientState(GL10.GL_TEXTURE_COORD_ARRAY); And here are the options I have enabled: gl.glShadeModel(GL10.GL_SMOOTH); //Enable Smooth Shading gl.glEnable(GL10.GL_DEPTH_TEST); //Enables Depth Testing gl.glDepthFunc(GL10.GL_LEQUAL); //The Type Of Depth Testing To Do gl.glEnable(GL10.GL_TEXTURE_2D); gl.glEnable(GL10.GL_BLEND); gl.glBlendFunc(GL10.GL_SRC_ALPHA,GL10.GL_ONE_MINUS_SRC_ALPHA); Edit: I just tried supplying a BitmapOptions to the BitmapFactory.decodeResource() call, but this doesn't seem to fix the issue, despite manually setting the same preferredconfig, density, and targetdensity. Edit2: As requested, here is a screenshot of the emulator working. The underlaying triangles are shown with a circle texture rendered onto it, the transparency is working because you can see the black background. Here is a shot of what the droid does with the exact same code on it: Edit3: Here are my BitmapOptions, updated the call above with how I am now calling the BitmapFactory, still the same results as below: sBitmapOptions.inPreferredConfig = Bitmap.Config.RGB_565; sBitmapOptions.inDensity = 160; sBitmapOptions.inTargetDensity = 160; sBitmapOptions.inScreenDensity = 160; sBitmapOptions.inDither = false; sBitmapOptions.inSampleSize = 1; sBitmapOptions.inScaled = false; Here are my vertices, texture coords, and indices: /** The initial vertex definition */ private static final float vertices[] = { -1.0f, -1.0f, 0.0f, 1.0f, -1.0f, 0.0f, -1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 0.0f }; /** The initial texture coordinates (u, v) */ private static final float texture[] = { //Mapping coordinates for the vertices 0.0f, 1.0f, 1.0f, 1.0f, 0.0f, 0.0f, 1.0f, 0.0f }; /** The initial indices definition */ private static final byte indices[] = { //Faces definition 0,1,3, 0,3,2 }; Is there anyway to dump the contents of the texture once it's been loaded into OpenGL ES? Maybe I can compare the emulator's loaded texture with the actual device's loaded texture? I did try with a different texture (the default android icon) and again, it works fine for the emulator but fails to render on the actual phone. Edit4: Tried switching around when I do texture loading. No luck. Tried using a constant offset of 0 to glGenTextures, no change. Is there something that I'm using that the emulator supports that the actual phone does not? Edit5: Per Ryan below, I resized my texture from 200x200 to 256x256, and the issue was NOT resolved. Edit: As requested, added the calls to glVertexPointer and glTexCoordPointer above. Also, here is the initialization of vertexBuffer, textureBuffer, and indexBuffer: ByteBuffer byteBuf = ByteBuffer.allocateDirect(vertices.length * 4); byteBuf.order(ByteOrder.nativeOrder()); vertexBuffer = byteBuf.asFloatBuffer(); vertexBuffer.put(vertices); vertexBuffer.position(0); byteBuf = ByteBuffer.allocateDirect(texture.length * 4); byteBuf.order(ByteOrder.nativeOrder()); textureBuffer = byteBuf.asFloatBuffer(); textureBuffer.put(texture); textureBuffer.position(0); indexBuffer = ByteBuffer.allocateDirect(indices.length); indexBuffer.put(indices); indexBuffer.position(0); loadGLTextures(gl, this.context);

Read the article

C++ SDL State Machine Segfault

- by user1602079

The code compiles and builds fine, but it immediately segfaults. I've looked at this for a while and have no idea why. Any help is appreciated. Thank you! Here's the code: main.cpp #include "SDL/SDL.h" #include "Globals.h" #include "Core.h" #include "GameStates.h" #include "Introduction.h" int main(int argc, char** args) { if(core.Initilize() == false) { SDL_Quit(); } while(core.desiredstate != core.Quit) { currentstate->EventHandling(); currentstate->Logic(); core.ChangeState(); currentstate->Render(); currentstate->Update(); } SDL_Quit(); } Core.h #ifndef CORE_H #define CORE_H #include "SDL/SDL.h" #include <string> class Core { public: SDL_Surface* Load(std::string filename); void ApplySurface(int X, int Y, SDL_Surface* source, SDL_Surface* destination); void SetState(int newstate); void ChangeState(); enum state { Intro, STATES_NULL, Quit }; int desiredstate, stateID; bool Initilize(); }; #endif Core.cpp #include "Core.h" #include "SDL/SDL.h" #include "Globals.h" #include "Introduction.h" #include <string> /* Initilizes SDL subsystems */ bool Core::Initilize() { //Inits subsystems, reutrns false upon error if(SDL_Init(SDL_INIT_EVERYTHING) == -1) { return false; } SDL_WM_SetCaption("Game", NULL); return true; } /* Loads surfaces and optimizes them */ SDL_Surface* Core::Load(std::string filename) { //The surface to be optimized SDL_Surface* original = SDL_LoadBMP(filename.c_str()); //The optimized surface SDL_Surface* optimized = NULL; //Optimizes the image if it loaded properly if(original != NULL) { optimized = SDL_DisplayFormat(original); SDL_FreeSurface(original); } else { //returns NULL upon error return NULL; } return optimized; } /* Blits surfaces */ void Core::ApplySurface(int X, int Y, SDL_Surface* source, SDL_Surface* destination) { //Stores the coordinates of the surface SDL_Rect offsets; offsets.x = X; offsets.y = Y; //Bits the surface if both surfaces are present if(source != NULL && destination != NULL) { SDL_BlitSurface(source, NULL, destination, &offsets); } } /* Sets desiredstate to newstate */ void Core::SetState(int newstate) { if(desiredstate != Quit) { desiredstate = newstate; } } /* Changes the game state */ void Core::ChangeState() { if(desiredstate != STATES_NULL && desiredstate != Quit) { delete currentstate; switch(desiredstate) { case Intro: currentstate = new Introduction(); break; } stateID = desiredstate; desiredstate = core.STATES_NULL; } } Globals.h #ifndef GLOBALS_H #define GLOBALS_H #include "SDL/SDL.h" #include "Core.h" #include "GameStates.h" extern SDL_Surface* screen; extern Core core; extern GameStates* currentstate; #endif Globals.cpp #include "Globals.h" #include "SDL/SDL.h" #include "GameStates.h" SDL_Surface* screen = SDL_SetVideoMode(640, 480, 32, SDL_SWSURFACE); Core core; GameStates* currentstate = NULL; GameStates.h #ifndef GAMESTATES_H #define GAMESTATES_H class GameStates { public: virtual void EventHandling() = 0; virtual void Logic() = 0; virtual void Render() = 0; virtual void Update() = 0; }; #endif Introduction.h #ifndef INTRODUCTION_H #define INTRODUCTION_H #include "GameStates.h" #include "Globals.h" class Introduction : public GameStates { public: Introduction(); private: void EventHandling(); void Logic(); void Render(); void Update(); ~Introduction(); SDL_Surface* test; }; #endif Introduction.cpp #include "SDL/SDL.h" #include "Core.h" #include "Globals.h" #include "Introduction.h" /* Loads all the assets */ Introduction::Introduction() { test = core.Load("test.bmp"); } void Introduction::EventHandling() { SDL_Event event; while(SDL_PollEvent(&event)) { switch(event.type) { case SDL_QUIT: core.SetState(core.Quit); break; } } } void Introduction::Logic() { //to be coded } void Introduction::Render() { core.ApplySurface(30, 30, test, screen); } void Introduction::Update() { SDL_Flip(screen); } Introduction::~Introduction() { SDL_FreeSurface(test); } Sorry if the formatting is a bit off... Having to put four spaces for it to be put into a code block offset it a bit. I ran it through gdb and this is what I got: Program received signal SIGSEGV, Segmentation fault. 0x0000000000400e46 in main () Which isn't incredibly useful... Any help is appreciated. Thank you!

Read the article

Windows using the DNS suffix search list on all lookups, even valid FQDNs. How to stop this?

- by RealityGone

When doing DNS lookups (specifically using nslookup, for some reason most things are not effected) Windows XP Pro SP3 is using the DNS suffix search list for every single one. Even for fully qualified domain names. For example I lookup "www.microsoft.com" but windows actually asks for "www.microsoft.com.eondream.com" (eondream.com is my primary domain). Now I can fix the issue by removing the Primary DNS suffix, but it seems to me that the DNS suffix search list should be for short, invalid names (where dots=0 or something). I'm sure I have a misconfiguration somewhere in windows but I don't know where. I've changed every option I can think of or find. Below is the output of ipconfig /all and nslookup (with debug & db2 enabled). This is using a static IP & (internal) DNS server. C:\ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : frayedlogic Primary Dns Suffix . . . . . . . : eondream.com Node Type . . . . . . . . . . . . : Unknown IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : eondream.com Ethernet adapter Wireless Network Connection: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Dell Wireless 1390 WLAN Mini-Card Physical Address. . . . . . . . . : 00-1B-FC-29-EB-6B Dhcp Enabled. . . . . . . . . . . : No IP Address. . . . . . . . . . . . : 192.168.13.32 Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 192.168.13.13 DNS Servers . . . . . . . . . . . : 192.168.19.19 C:\nslookup Default Server: shardik.eondream.com Address: 192.168.19.19 set debug set db2 www.microsoft.com Server: shardik.eondream.com Address: 192.168.19.19 ------------ Got answer: HEADER: opcode = QUERY, id = 2, rcode = NOERROR header flags: response, want recursion, recursion avail. questions = 1, answers = 1, authority records = 0, additional = 0 QUESTIONS: www.microsoft.com.eondream.com, type = A, class = IN ANSWERS: - www.microsoft.com.eondream.com internet address = 208.69.36.132 ttl = 0 (0 secs) ------------ Non-authoritative answer: Name: www.microsoft.com.eondream.com Address: 208.69.36.132 (Note: it resolves to that IP because I use the opendns service and that is their suggestion page or whatever you want to call it) If I am reading the nslookup output correctly then it is not a problem with my DNS server because windows is actually asking for the incorrect domain.

Read the article

Inconsistent results in R with RNetCDF - why?

- by sarcozona

I am having trouble extracting data from NetCDF data files using RNetCDF. The data files each have 3 dimensions (longitude, latitude, and a date) and 3 variables (latitude, longitude, and a climate variable). There are four datasets, each with a different climate variable. Here is some of the output from print.nc(p8m.tmax) for clarity. The other datasets are identical except for the specific climate variable. dimensions: month = UNLIMITED ; // (1368 currently) lat = 3105 ; lon = 7025 ; variables: float lat(lat) ; lat:long_name = "latitude" ; lat:standard_name = "latitude" ; lat:units = "degrees_north" ; float lon(lon) ; lon:long_name = "longitude" ; lon:standard_name = "longitude" ; lon:units = "degrees_east" ; short tmax(lon, lat, month) ; tmax:missing_value = -9999 ; tmax:_FillValue = -9999 ; tmax:units = "degree_celsius" ; tmax:scale_factor = 0.01 ; tmax:valid_min = -5000 ; tmax:valid_max = 6000 ; I am getting behavior I don't understand when I use the var.get.nc function from the RNetCDF package. For example, when I attempt to extract 82 values beginning at stval from the maximum temperature data (p8m.tmax <- open.nc(tmaxdataset.nc)) with > var.get.nc(p8m.tmax,'tmax', start=c(lon_val, lat_val, stval),count=c(1,1,82)) (where lon_val and lat_val specify the location in the dataset of the coordinates I'm interested in and stval is stval is set to which(time_vec==200201), which in this case equaled 1285.) I get Error: Invalid argument But after successfully extracting 80 and 81 values > var.get.nc(p8m.tmax,'tmax', start=c(lon_val, lat_val, stval),count=c(1,1,80)) > var.get.nc(p8m.tmax,'tmax', start=c(lon_val, lat_val, stval),count=c(1,1,81)) the command with 82 works: > var.get.nc(p8m.tmax,'tmax', start=c(lon_val, lat_val, stval),count=c(1,1,82)) [1] 444 866 1063 ... [output snipped] The same problem occurs in the identically structured tmin file, but at 36 instead of 82: > var.get.nc(p8m.tmin,'tmin', start=c(lon_val, lat_val, stval),count=c(1,1,36)) produces Error: Invalid argument But after repeating with counts of 30, 31, etc > var.get.nc(p8m.tmin,'tmin', start=c(lon_val, lat_val, stval), count=c(1,1,36)) works. These examples make it seem like the function is failing at the last count, but that actually isn't the case. In the first example, var.get.nc gave Error: Invalid argument after I asked for 84 values. I then narrowed the failure down to the 82nd count by varying the starting point in the dataset and asking for only 1 value at a time. The particular number the problem occurs at also varies. I can close and reopen the dataset and have the problem occur at a different location. In the particular examples above, lon_val and lat_val are 1595 and 1751, respectively, identifying the location in the dataset along the lat and lon dimensions for the latitude and longitude I'm interested in. The 1595th latitude and 1751th longitude are not the problem, however. The problem occurs with all other latitude and longitudes I've tried. Varying the starting location in the dataset along the climate variable dimension (stval) and/or specifying it different (as a number in the command instead of the object stval) also does not fix the problem. This problem doesn't always occur. I can run identical code three times in a row (clearing all objects in between runs) and get a different outcome each time. The first run may choke on the 7th entry I'm trying to get, the second might work fine, and the third run might choke on the 83rd entry. I'm absolutely baffled by such inconsistent behavior. The open.nc function has also started to fail with the same Error: Invalid argument. Like the var.get.nc problems, it also occurs inconsistently. Does anyone know what causes the initial failure to extract the variable? And how I might prevent it? Could have to do with the size of the data files (~60GB each) and/or the fact that I'm accessing them through networked drives? This was also asked here: https://stat.ethz.ch/pipermail/r-help/2011-June/281233.html > sessionInfo() R version 2.13.0 (2011-04-13) Platform: i386-pc-mingw32/i386 (32-bit) locale: [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 [3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C [5] LC_TIME=English_United States.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] reshape_0.8.4 plyr_1.5.2 RNetCDF_1.5.2-2 loaded via a namespace (and not attached): [1] tools_2.13.0

Read the article

How to stop rendering invisible faces

- by TheMorfeus

I am making a voxel-based game, and for needs of it, i am creating a block rendering engine. Point is, that i need to generate lots of cubes. Every time i render more than 16x16x16 chunk of theese blocks, my FPS is dropped down hardly, because it renders all 6 faces of all of theese cubes. THat's 24 576 quads, and i dont want that. So, my question is, How to stop rendering vertices(or quads) that are not visible, and therefore increase performance of my game? Here is class for rendering of a block: public void renderBlock(int posx, int posy, int posz) { try{ //t.bind(); glEnable(GL_CULL_FACE); glCullFace(GL_BACK);// or even GL_FRONT_AND_BACK */); glPushMatrix(); GL11.glTranslatef((2*posx+0.5f),(2*posy+0.5f),(2*posz+0.5f)); // Move Right 1.5 Units And Into The Screen 6.0 GL11.glRotatef(rquad,1.0f,1.0f,1.0f); glBegin(GL_QUADS); // Draw A Quad GL11.glColor3f(0.5f, 0.4f, 0.4f); // Set The Color To Green GL11.glTexCoord2f(0,0); GL11.glVertex3f( 1f, 1f,-1f); // Top Right Of The Quad (Top) GL11.glTexCoord2f(1,0); GL11.glVertex3f(-1f, 1f,-1f); // Top Left Of The Quad (Top) GL11.glTexCoord2f(1,1); GL11.glVertex3f(-1f, 1f, 1f); // Bottom Left Of The Quad (Top) GL11.glTexCoord2f(0,1); GL11.glVertex3f( 1f, 1f, 1f); // Bottom Right Of The Quad (Top) //GL11.glColor3f(1.2f,0.5f,0.9f); // Set The Color To Orange GL11.glTexCoord2f(0,0); GL11.glVertex3f( 1f,-1f, 1f); // Top Right Of The Quad (Bottom) GL11.glTexCoord2f(0,1); GL11.glVertex3f(-1f,-1f, 1f); // Top Left Of The Quad (Bottom) GL11.glTexCoord2f(1,1); GL11.glVertex3f(-1f,-1f,-1f); // Bottom Left Of The Quad (Bottom) GL11.glTexCoord2f(1,0); GL11.glVertex3f( 1f,-1f,-1f); // Bottom Right Of The Quad (Bottom) //GL11.glColor3f(1.0f,0.0f,0.0f); // Set The Color To Red GL11.glTexCoord2f(0,0); GL11.glVertex3f( 1f, 1f, 1f); // Top Right Of The Quad (Front) GL11.glTexCoord2f(1,0); GL11.glVertex3f(-1f, 1f, 1f); // Top Left Of The Quad (Front) GL11.glTexCoord2f(1,1); GL11.glVertex3f(-1f,-1f, 1f); // Bottom Left Of The Quad (Front) GL11.glTexCoord2f(0,1); GL11.glVertex3f( 1f,-1f, 1f); // Bottom Right Of The Quad (Front) //GL11.glColor3f(1f,0.5f,0.0f); // Set The Color To Yellow GL11.glTexCoord2f(0,0); GL11.glVertex3f( 1f,-1f,-1f); // Bottom Left Of The Quad (Back) GL11.glTexCoord2f(1,0); GL11.glVertex3f(-1f,-1f,-1f); // Bottom Right Of The Quad (Back) GL11.glTexCoord2f(1,1); GL11.glVertex3f(-1f, 1f,-1f); // Top Right Of The Quad (Back) GL11.glTexCoord2f(0,1); GL11.glVertex3f( 1f, 1f,-1f); // Top Left Of The Quad (Back) //GL11.glColor3f(0.0f,0.0f,0.3f); // Set The Color To Blue GL11.glTexCoord2f(0,1); GL11.glVertex3f(-1f, 1f, 1f); // Top Right Of The Quad (Left) GL11.glTexCoord2f(1,1); GL11.glVertex3f(-1f, 1f,-1f); // Top Left Of The Quad (Left) GL11.glTexCoord2f(1,0); GL11.glVertex3f(-1f,-1f,-1f); // Bottom Left Of The Quad (Left) GL11.glTexCoord2f(0,0); GL11.glVertex3f(-1f,-1f, 1f); // Bottom Right Of The Quad (Left) //GL11.glColor3f(0.5f,0.0f,0.5f); // Set The Color To Violet GL11.glTexCoord2f(0,0); GL11.glVertex3f( 1f, 1f,-1f); // Top Right Of The Quad (Right) GL11.glTexCoord2f(1,0); GL11.glVertex3f( 1f, 1f, 1f); // Top Left Of The Quad (Right) GL11.glTexCoord2f(1,1); GL11.glVertex3f( 1f,-1f, 1f); // Bottom Left Of The Quad (Right) GL11.glTexCoord2f(0,1); GL11.glVertex3f( 1f,-1f,-1f); // Bottom Right Of The Quad (Right) //rquad+=0.0001f; glEnd(); glPopMatrix(); }catch(NullPointerException t){t.printStackTrace(); System.out.println("rendering block failed");} } Here is code that renders them: private void render() { GL11.glClear(GL11.GL_COLOR_BUFFER_BIT|GL11.GL_DEPTH_BUFFER_BIT); for(int y=0; y<32; y++){ for(int x=0; x<16; x++){ for(int z=0; z<16; z++) { b.renderBlock(x, y, z); } } } }

Read the article

Building my first Javascript Application (jQuery), struggling on something

- by Jason Wells

I'd really appreciate recommendations on the most efficient way to approach this. I'm building a simple javascript application which displays a list of records and allows the user to edit a record by clicking an "Edit" link in the records row. The user also can click the "Add" link to pop open a dialog allowing them to add a new record. Here's a working prototype of this: http://jsfiddle.net/FfRcG/ You'll note if you click "Edit" a dialog pops up with some canned values. And, if you click "Add", a dialog pops up with empty values. I need help on how to approach two problems I believe we need to pass our index to our edit dialog and reference the values within the JSON, but I am unsure how to pass the index when the user clicks edit. It bothers me that the Edit and Add div contents are so similiar (Edit just pre populates the values). I feel like there is a more efficient way of doing this but am at a loss. Here is my code for reference $(document).ready( function(){ // Our JSON (This would actually be coming from an AJAX database call) people = { "COLUMNS":["DATEMODIFIED", "NAME","AGE"], "DATA":[ ["9/6/2012", "Person 1","32"], ["9/5/2012","Person 2","23"] ] } // Here we loop over our JSON and build our HTML (Will refactor to use templating eventually) members = people.DATA; var newcontent = '<table width=50%><tr><td>date</td><td>name</td><td>age</td><td></td></tr>'; for(var i=0;i<members.length;i++) { newcontent+= '<tr id="member'+i+'"><td>' + members[i][0] + '</td>'; newcontent+= '<td>' + members[i][1] + '</td>'; newcontent+= '<td>' + members[i][2] + '</td>'; newcontent+= '<td><a href="#" class="edit" id=edit'+i+'>Edit</a></td><td>'; } newcontent += "</table>"; $("#result").html(newcontent); // Bind a dialog to the edit link $(".edit").click( function(){ // Trigger our dialog to open $("#edit").dialog("open"); // Not sure the most efficient way to change our dialog field values $("#name").val() // ??? alert($()); return false; }); // Bind a dialog to the add link $(".edit").click( function(){ // Trigger our dialog to open $("#add").dialog("open"); return false; }); // Bind a dialog to our edit DIV $("#edit").dialog(); // Bind a dialog to our add DIV $("#add").dialog(); }); And here's the HTML <h1>People</h1> <a href="#" class="add">Add a new person</a>  <div id="result"></div>  <div id="edit"> <form> <p>Name:<br/><input type="text" id="name" value="foo"></p> <p>Age:<br/><input type="text" id="age" value="33"></p> <input type="submit" value="Save" id="submitEdit"> </form> </div>  <div id="add"> <form> <p>Name:<br/><input type="text" id="name"></p> <p>Age:<br/><input type="text" id="age"></p> <input type="submit" value="Save" id="submitEdit"> </form> </div>

Read the article

Creating a dynamic, extensible C# Expando Object

- by Rick Strahl

I love dynamic functionality in a strongly typed language because it offers us the best of both worlds. In C# (or any of the main .NET languages) we now have the dynamic type that provides a host of dynamic features for the static C# language. One place where I've found dynamic to be incredibly useful is in building extensible types or types that expose traditionally non-object data (like dictionaries) in easier to use and more readable syntax. I wrote about a couple of these for accessing old school ADO.NET DataRows and DataReaders more easily for example. These classes are dynamic wrappers that provide easier syntax and auto-type conversions which greatly simplifies code clutter and increases clarity in existing code. ExpandoObject in .NET 4.0 Another great use case for dynamic objects is the ability to create extensible objects - objects that start out with a set of static members and then can add additional properties and even methods dynamically. The .NET 4.0 framework actually includes an ExpandoObject class which provides a very dynamic object that allows you to add properties and methods on the fly and then access them again. For example with ExpandoObject you can do stuff like this:dynamic expand = new ExpandoObject(); expand.Name = "Rick"; expand.HelloWorld = (Func<string, string>) ((string name) => { return "Hello " + name; }); Console.WriteLine(expand.Name); Console.WriteLine(expand.HelloWorld("Dufus")); Internally ExpandoObject uses a Dictionary like structure and interface to store properties and methods and then allows you to add and access properties and methods easily. As cool as ExpandoObject is it has a few shortcomings too: It's a sealed type so you can't use it as a base class It only works off 'properties' in the internal Dictionary - you can't expose existing type data It doesn't serialize to XML or with DataContractSerializer/DataContractJsonSerializer Expando - A truly extensible Object ExpandoObject is nice if you just need a dynamic container for a dictionary like structure. However, if you want to build an extensible object that starts out with a set of strongly typed properties and then allows you to extend it, ExpandoObject does not work because it's a sealed class that can't be inherited. I started thinking about this very scenario for one of my applications I'm building for a customer. In this system we are connecting to various different user stores. Each user store has the same basic requirements for username, password, name etc. But then each store also has a number of extended properties that is available to each application. In the real world scenario the data is loaded from the database in a data reader and the known properties are assigned from the known fields in the database. All unknown fields are then 'added' to the expando object dynamically. In the past I've done this very thing with a separate property - Properties - just like I do for this class. But the property and dictionary syntax is not ideal and tedious to work with. I started thinking about how to represent these extra property structures. One way certainly would be to add a Dictionary, or an ExpandoObject to hold all those extra properties. But wouldn't it be nice if the application could actually extend an existing object that looks something like this as you can with the Expando object:public class User : Westwind.Utilities.Dynamic.Expando { public string Email { get; set; } public string Password { get; set; } public string Name { get; set; } public bool Active { get; set; } public DateTime? ExpiresOn { get; set; } } and then simply start extending the properties of this object dynamically? Using the Expando object I describe later you can now do the following:[TestMethod] public void UserExampleTest() { var user = new User(); // Set strongly typed properties user.Email = "[email protected]"; user.Password = "nonya123"; user.Name = "Rickochet"; user.Active = true; // Now add dynamic properties dynamic duser = user; duser.Entered = DateTime.Now; duser.Accesses = 1; // you can also add dynamic props via indexer user["NickName"] = "AntiSocialX"; duser["WebSite"] = "http://www.west-wind.com/weblog"; // Access strong type through dynamic ref Assert.AreEqual(user.Name,duser.Name); // Access strong type through indexer Assert.AreEqual(user.Password,user["Password"]); // access dyanmically added value through indexer Assert.AreEqual(duser.Entered,user["Entered"]); // access index added value through dynamic Assert.AreEqual(user["NickName"],duser.NickName); // loop through all properties dynamic AND strong type properties (true) foreach (var prop in user.GetProperties(true)) { object val = prop.Value; if (val == null) val = "null"; Console.WriteLine(prop.Key + ": " + val.ToString()); } } As you can see this code somewhat blurs the line between a static and dynamic type. You start with a strongly typed object that has a fixed set of properties. You can then cast the object to dynamic (as I discussed in my last post) and add additional properties to the object. You can also use an indexer to add dynamic properties to the object. To access the strongly typed properties you can use either the strongly typed instance, the indexer or the dynamic cast of the object. Personally I think it's kinda cool to have an easy way to access strongly typed properties by string which can make some data scenarios much easier. To access the 'dynamically added' properties you can use either the indexer on the strongly typed object, or property syntax on the dynamic cast. Using the dynamic type allows all three modes to work on both strongly typed and dynamic properties. Finally you can iterate over all properties, both dynamic and strongly typed if you chose. Lots of flexibility. Note also that by default the Expando object works against the (this) instance meaning it extends the current object. You can also pass in a separate instance to the constructor in which case that object will be used to iterate over to find properties rather than this. Using this approach provides some really interesting functionality when use the dynamic type. To use this we have to add an explicit constructor to the Expando subclass:public class User : Westwind.Utilities.Dynamic.Expando { public string Email { get; set; } public string Password { get; set; } public string Name { get; set; } public bool Active { get; set; } public DateTime? ExpiresOn { get; set; } public User() : base() { } // only required if you want to mix in seperate instance public User(object instance) : base(instance) { } } to allow the instance to be passed. When you do you can now do:[TestMethod] public void ExpandoMixinTest() { // have Expando work on Addresses var user = new User( new Address() ); // cast to dynamicAccessToPropertyTest dynamic duser = user; // Set strongly typed properties duser.Email = "[email protected]"; user.Password = "nonya123"; // Set properties on address object duser.Address = "32 Kaiea"; //duser.Phone = "808-123-2131"; // set dynamic properties duser.NonExistantProperty = "This works too"; // shows default value Address.Phone value Console.WriteLine(duser.Phone); } Using the dynamic cast in this case allows you to access *three* different 'objects': The strong type properties, the dynamically added properties in the dictionary and the properties of the instance passed in! Effectively this gives you a way to simulate multiple inheritance (which is scary - so be very careful with this, but you can do it). How Expando works Behind the scenes Expando is a DynamicObject subclass as I discussed in my last post. By implementing a few of DynamicObject's methods you can basically create a type that can trap 'property missing' and 'method missing' operations. When you access a non-existant property a known method is fired that our code can intercept and provide a value for. Internally Expando uses a custom dictionary implementation to hold the dynamic properties you might add to your expandable object. Let's look at code first. The code for the Expando type is straight forward and given what it provides relatively short. Here it is.using System; using System.Collections.Generic; using System.Linq; using System.Dynamic; using System.Reflection; namespace Westwind.Utilities.Dynamic { /// <summary> /// Class that provides extensible properties and methods. This /// dynamic object stores 'extra' properties in a dictionary or /// checks the actual properties of the instance. /// /// This means you can subclass this expando and retrieve either /// native properties or properties from values in the dictionary. /// /// This type allows you three ways to access its properties: /// /// Directly: any explicitly declared properties are accessible /// Dynamic: dynamic cast allows access to dictionary and native properties/methods /// Dictionary: Any of the extended properties are accessible via IDictionary interface /// </summary> [Serializable] public class Expando : DynamicObject, IDynamicMetaObjectProvider { /// <summary> /// Instance of object passed in /// </summary> object Instance; /// <summary> /// Cached type of the instance /// </summary> Type InstanceType; PropertyInfo[] InstancePropertyInfo { get { if (_InstancePropertyInfo == null && Instance != null) _InstancePropertyInfo = Instance.GetType().GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.DeclaredOnly); return _InstancePropertyInfo; } } PropertyInfo[] _InstancePropertyInfo; /// <summary> /// String Dictionary that contains the extra dynamic values /// stored on this object/instance /// </summary> /// <remarks>Using PropertyBag to support XML Serialization of the dictionary</remarks> public PropertyBag Properties = new PropertyBag(); //public Dictionary<string,object> Properties = new Dictionary<string, object>(); /// <summary> /// This constructor just works off the internal dictionary and any /// public properties of this object. /// /// Note you can subclass Expando. /// </summary> public Expando() { Initialize(this); } /// <summary> /// Allows passing in an existing instance variable to 'extend'. /// </summary> /// <remarks> /// You can pass in null here if you don't want to /// check native properties and only check the Dictionary! /// </remarks> /// <param name="instance"></param> public Expando(object instance) { Initialize(instance); } protected virtual void Initialize(object instance) { Instance = instance; if (instance != null) InstanceType = instance.GetType(); } /// <summary> /// Try to retrieve a member by name first from instance properties /// followed by the collection entries. /// </summary> /// <param name="binder"></param> /// <param name="result"></param> /// <returns></returns> public override bool TryGetMember(GetMemberBinder binder, out object result) { result = null; // first check the Properties collection for member if (Properties.Keys.Contains(binder.Name)) { result = Properties[binder.Name]; return true; } // Next check for Public properties via Reflection if (Instance != null) { try { return GetProperty(Instance, binder.Name, out result); } catch { } } // failed to retrieve a property result = null; return false; } /// <summary> /// Property setter implementation tries to retrieve value from instance /// first then into this object /// </summary> /// <param name="binder"></param> /// <param name="value"></param> /// <returns></returns> public override bool TrySetMember(SetMemberBinder binder, object value) { // first check to see if there's a native property to set if (Instance != null) { try { bool result = SetProperty(Instance, binder.Name, value); if (result) return true; } catch { } } // no match - set or add to dictionary Properties[binder.Name] = value; return true; } /// <summary> /// Dynamic invocation method. Currently allows only for Reflection based /// operation (no ability to add methods dynamically). /// </summary> /// <param name="binder"></param> /// <param name="args"></param> /// <param name="result"></param> /// <returns></returns> public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) { if (Instance != null) { try { // check instance passed in for methods to invoke if (InvokeMethod(Instance, binder.Name, args, out result)) return true; } catch { } } result = null; return false; } /// <summary> /// Reflection Helper method to retrieve a property /// </summary> /// <param name="instance"></param> /// <param name="name"></param> /// <param name="result"></param> /// <returns></returns> protected bool GetProperty(object instance, string name, out object result) { if (instance == null) instance = this; var miArray = InstanceType.GetMember(name, BindingFlags.Public | BindingFlags.GetProperty | BindingFlags.Instance); if (miArray != null && miArray.Length > 0) { var mi = miArray[0]; if (mi.MemberType == MemberTypes.Property) { result = ((PropertyInfo)mi).GetValue(instance,null); return true; } } result = null; return false; } /// <summary> /// Reflection helper method to set a property value /// </summary> /// <param name="instance"></param> /// <param name="name"></param> /// <param name="value"></param> /// <returns></returns> protected bool SetProperty(object instance, string name, object value) { if (instance == null) instance = this; var miArray = InstanceType.GetMember(name, BindingFlags.Public | BindingFlags.SetProperty | BindingFlags.Instance); if (miArray != null && miArray.Length > 0) { var mi = miArray[0]; if (mi.MemberType == MemberTypes.Property) { ((PropertyInfo)mi).SetValue(Instance, value, null); return true; } } return false; } /// <summary> /// Reflection helper method to invoke a method /// </summary> /// <param name="instance"></param> /// <param name="name"></param> /// <param name="args"></param> /// <param name="result"></param> /// <returns></returns> protected bool InvokeMethod(object instance, string name, object[] args, out object result) { if (instance == null) instance = this; // Look at the instanceType var miArray = InstanceType.GetMember(name, BindingFlags.InvokeMethod | BindingFlags.Public | BindingFlags.Instance); if (miArray != null && miArray.Length > 0) { var mi = miArray[0] as MethodInfo; result = mi.Invoke(Instance, args); return true; } result = null; return false; } /// <summary> /// Convenience method that provides a string Indexer /// to the Properties collection AND the strongly typed /// properties of the object by name. /// /// // dynamic /// exp["Address"] = "112 nowhere lane"; /// // strong /// var name = exp["StronglyTypedProperty"] as string; /// </summary> /// <remarks> /// The getter checks the Properties dictionary first /// then looks in PropertyInfo for properties. /// The setter checks the instance properties before /// checking the Properties dictionary. /// </remarks> /// <param name="key"></param> /// /// <returns></returns> public object this[string key] { get { try { // try to get from properties collection first return Properties[key]; } catch (KeyNotFoundException ex) { // try reflection on instanceType object result = null; if (GetProperty(Instance, key, out result)) return result; // nope doesn't exist throw; } } set { if (Properties.ContainsKey(key)) { Properties[key] = value; return; } // check instance for existance of type first var miArray = InstanceType.GetMember(key, BindingFlags.Public | BindingFlags.GetProperty); if (miArray != null && miArray.Length > 0) SetProperty(Instance, key, value); else Properties[key] = value; } } /// <summary> /// Returns and the properties of /// </summary> /// <param name="includeProperties"></param> /// <returns></returns> public IEnumerable<KeyValuePair<string,object>> GetProperties(bool includeInstanceProperties = false) { if (includeInstanceProperties && Instance != null) { foreach (var prop in this.InstancePropertyInfo) yield return new KeyValuePair<string, object>(prop.Name, prop.GetValue(Instance, null)); } foreach (var key in this.Properties.Keys) yield return new KeyValuePair<string, object>(key, this.Properties[key]); } /// <summary> /// Checks whether a property exists in the Property collection /// or as a property on the instance /// </summary> /// <param name="item"></param> /// <returns></returns> public bool Contains(KeyValuePair<string, object> item, bool includeInstanceProperties = false) { bool res = Properties.ContainsKey(item.Key); if (res) return true; if (includeInstanceProperties && Instance != null) { foreach (var prop in this.InstancePropertyInfo) { if (prop.Name == item.Key) return true; } } return false; } } } Although the Expando class supports an indexer, it doesn't actually implement IDictionary or even IEnumerable. It only provides the indexer and Contains() and GetProperties() methods, that work against the Properties dictionary AND the internal instance. The reason for not implementing IDictionary is that a) it doesn't add much value since you can access the Properties dictionary directly and that b) I wanted to keep the interface to class very lean so that it can serve as an entity type if desired. Implementing these IDictionary (or even IEnumerable) causes LINQ extension methods to pop up on the type which obscures the property interface and would only confuse the purpose of the type. IDictionary and IEnumerable are also problematic for XML and JSON Serialization - the XML Serializer doesn't serialize IDictionary<string,object>, nor does the DataContractSerializer. The JavaScriptSerializer does serialize, but it treats the entire object like a dictionary and doesn't serialize the strongly typed properties of the type, only the dictionary values which is also not desirable. Hence the decision to stick with only implementing the indexer to support the user["CustomProperty"] functionality and leaving iteration functions to the publicly exposed Properties dictionary. Note that the Dictionary used here is a custom PropertyBag class I created to allow for serialization to work. One important aspect for my apps is that whatever custom properties get added they have to be accessible to AJAX clients since the particular app I'm working on is a SIngle Page Web app where most of the Web access is through JSON AJAX calls. PropertyBag can serialize to XML and one way serialize to JSON using the JavaScript serializer (not the DCS serializers though). The key components that make Expando work in this code are the Properties Dictionary and the TryGetMember() and TrySetMember() methods. The Properties collection is public so if you choose you can explicitly access the collection to get better performance or to manipulate the members in internal code (like loading up dynamic values form a database). Notice that TryGetMember() and TrySetMember() both work against the dictionary AND the internal instance to retrieve and set properties. This means that user["Name"] works against native properties of the object as does user["Name"] = "RogaDugDog". What's your Use Case? This is still an early prototype but I've plugged it into one of my customer's applications and so far it's working very well. The key features for me were the ability to easily extend the type with values coming from a database and exposing those values in a nice and easy to use manner. I'm also finding that using this type of object for ViewModels works very well to add custom properties to view models. I suspect there will be lots of uses for this - I've been using the extra dictionary approach to extensibility for years - using a dynamic type to make the syntax cleaner is just a bonus here. What can you think of to use this for? Resources Source Code and Tests (GitHub) Also integrated in Westwind.Utilities of the West Wind Web Toolkit West Wind Utilities NuGet© Rick Strahl, West Wind Technologies, 2005-2012Posted in CSharp .NET Dynamic Types Tweet !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs"); (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

Read the article

Oracle Coherence, Split-Brain and Recovery Protocols In Detail

- by Ricardo Ferreira

This article provides a high level conceptual overview of Split-Brain scenarios in distributed systems. It will focus on a specific example of cluster communication failure and recovery in Oracle Coherence. This includes a discussion on the witness protocol (used to remove failed cluster members) and the panic protocol (used to resolve Split-Brain scenarios). Note that the removal of cluster members does not necessarily indicate a Split-Brain condition. Oracle Coherence does not (and cannot) detect a Split-Brain as it occurs, the condition is only detected when cluster members that previously lost contact with each other regain contact. Cluster Topology and Configuration In order to create an good didactic for the article, let's assume a cluster topology and configuration. In this example we have a six member cluster, consisting of one JVM on each physical machine. The member IDs are as follows: Member ID IP Address 1 10.149.155.76 2 10.149.155.77 3 10.149.155.236 4 10.149.155.75 5 10.149.155.79 6 10.149.155.78 Members 1, 2, and 3 are connected to a switch, and members 4, 5, and 6 are connected to a second switch. There is a link between the two switches, which provides network connectivity between all of the machines. Member 1 is the first member to join this cluster, thus making it the senior member. Member 6 is the last member to join this cluster. Here is a log snippet from Member 6 showing the complete member set: 2010-02-26 15:27:57.390/3.062 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=main, member=6): Started DefaultCacheServer... SafeCluster: Name=cluster:0xDDEB Group{Address=224.3.5.3, Port=35465, TTL=4} MasterMemberSet ( ThisMember=Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) OldestMember=Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) ActualMemberSet=MemberSet(Size=6, BitSetCount=2 Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) Member(Id=2, Timestamp=2010-02-26 15:27:17.847, Address=10.149.155.77:8088, MachineId=1101, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:296, Role=CoherenceServer) Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer) Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) Member(Id=5, Timestamp=2010-02-26 15:27:49.095, Address=10.149.155.79:8088, MachineId=1103, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:3229, Role=CoherenceServer) Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) ) RecycleMillis=120000 RecycleSet=MemberSet(Size=0, BitSetCount=0 ) ) At approximately 15:30, the connection between the two switches is severed: Thirty seconds later (the default packet timeout in development mode) the logs indicate communication failures across the cluster. In this example, the communication failure was caused by a network failure. In a production setting, this type of communication failure can have many root causes, including (but not limited to) network failures, excessive GC, high CPU utilization, swapping/virtual memory, and exceeding maximum network bandwidth. In addition, this type of failure is not necessarily indicative of a split brain. Any communication failure will be logged in this fashion. Member 2 logs a communication failure with Member 5: 2010-02-26 15:30:32.638/196.928 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=PacketPublisher, member=2): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=5, Timestamp=2010-02-26 15:27:49.095, Address=10.149.155.79:8088, MachineId=1103, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:3229, Role=CoherenceServer) by MemberSet(Size=2, BitSetCount=2 Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) ) The Coherence clustering protocol (TCMP) is a reliable transport mechanism built on UDP. In order for the protocol to be reliable, it requires an acknowledgement (ACK) for each packet delivered. If a packet fails to be acknowledged within the configured timeout period, the Coherence cluster member will log a packet timeout (as seen in the log message above). When this occurs, the cluster member will consult with other members to determine who is at fault for the communication failure. If the witness members agree that the suspect member is at fault, the suspect is removed from the cluster. If the witnesses unanimously disagree, the accuser is removed. This process is known as the witness protocol. Since Member 2 cannot communicate with Member 5, it selects two witnesses (Members 1 and 4) to determine if the communication issue is with Member 5 or with itself (Member 2). However, Member 4 is on the switch that is no longer accessible by Members 1, 2 and 3; thus a packet timeout for member 4 is recorded as well: 2010-02-26 15:30:35.648/199.938 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=PacketPublisher, member=2): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) by MemberSet(Size=2, BitSetCount=2 Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) ) Member 1 has the ability to confirm the departure of member 4, however Member 6 cannot as it is also inaccessible. At the same time, Member 3 sends a request to remove Member 6, which is followed by a report from Member 3 indicating that Member 6 has departed the cluster: 2010-02-26 15:30:35.706/199.996 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=2): MemberLeft request for Member 6 received from Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer) 2010-02-26 15:30:35.709/199.999 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=2): MemberLeft notification for Member 6 received from Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer) The log for Member 3 determines how Member 6 departed the cluster: 2010-02-26 15:30:35.161/191.694 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=PacketPublisher, member=3): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) by MemberSet(Size=2, BitSetCount=2 Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) Member(Id=2, Timestamp=2010-02-26 15:27:17.847, Address=10.149.155.77:8088, MachineId=1101, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:296, Role=CoherenceServer) ) 2010-02-26 15:30:35.165/191.698 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=Cluster, member=3): Member departure confirmed by MemberSet(Size=2, BitSetCount=2 Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) Member(Id=2, Timestamp=2010-02-26 15:27:17.847, Address=10.149.155.77:8088, MachineId=1101, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:296, Role=CoherenceServer) ); removing Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) In this case, Member 3 happened to select two witnesses that it still had connectivity with (Members 1 and 2) thus resulting in a simple decision to remove Member 6. Given the departure of Member 6, Member 2 is left with a single witness to confirm the departure of Member 4: 2010-02-26 15:30:35.713/200.003 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=Cluster, member=2): Member departure confirmed by MemberSet(Size=1, BitSetCount=2 Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) ); removing Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) In the meantime, Member 4 logs a missing heartbeat from the senior member. This message is also logged on Members 5 and 6. 2010-02-26 15:30:07.906/150.453 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=PacketListenerN, member=4): Scheduled senior member heartbeat is overdue; rejoining multicast group. Next, Member 4 logs a TcpRing failure with Member 2, thus resulting in the termination of Member 2: 2010-02-26 15:30:21.421/163.968 Oracle Coherence GE 3.5.3/465p2 <D4> (thread=Cluster, member=4): TcpRing: Number of socket exceptions exceeded maximum; last was "java.net.SocketTimeoutException: connect timed out"; removing the member: 2 For quick process termination detection, Oracle Coherence utilizes a feature called TcpRing which is a sparse collection of TCP/IP-based connections between different members in the cluster. Each member in the cluster is connected to at least one other member, which (if at all possible) is running on a different physical box. This connection is not used for any data transfer, only heartbeat communications are sent once a second per each link. If a certain number of exceptions are thrown while trying to re-establish a connection, the member throwing the exceptions is removed from the cluster. Member 5 logs a packet timeout with Member 3 and cites witnesses Members 4 and 6: 2010-02-26 15:30:29.791/165.037 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=PacketPublisher, member=5): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer) by MemberSet(Size=2, BitSetCount=2 Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) ) 2010-02-26 15:30:29.798/165.044 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=Cluster, member=5): Member departure confirmed by MemberSet(Size=2, BitSetCount=2 Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) Member(Id=6, Timestamp=2010-02-26 15:27:58.635, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) ); removing Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer) Eventually we are left with two distinct clusters consisting of Members 1, 2, 3 and Members 4, 5, 6, respectively. In the latter cluster, Member 4 is promoted to senior member. The connection between the two switches is restored at 15:33. Upon the restoration of the connection, the cluster members immediately receive cluster heartbeats from the two senior members. In the case of Members 1, 2, and 3, the following is logged: 2010-02-26 15:33:14.970/369.066 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=Cluster, member=1): The member formerly known as Member(Id=4, Timestamp=2010-02-26 15:30:35.341, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) has been forcefully evicted from the cluster, but continues to emit a cluster heartbeat; henceforth, the member will be shunned and its messages will be ignored. Likewise for Members 4, 5, and 6: 2010-02-26 15:33:14.343/336.890 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=Cluster, member=4): The member formerly known as Member(Id=1, Timestamp=2010-02-26 15:30:31.64, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) has been forcefully evicted from the cluster, but continues to emit a cluster heartbeat; henceforth, the member will be shunned and its messages will be ignored. This message indicates that a senior heartbeat is being received from members that were previously removed from the cluster, in other words, something that should not be possible. For this reason, the recipients of these messages will initially ignore them. After several iterations of these messages, the existence of multiple clusters is acknowledged, thus triggering the panic protocol to reconcile this situation. When the presence of more than one cluster (i.e. Split-Brain) is detected by a Coherence member, the panic protocol is invoked in order to resolve the conflicting clusters and consolidate into a single cluster. The protocol consists of the removal of smaller clusters until there is one cluster remaining. In the case of equal size clusters, the one with the older Senior Member will survive. Member 1, being the oldest member, initiates the protocol: 2010-02-26 15:33:45.970/400.066 Oracle Coherence GE 3.5.3/465p2 <Warning> (thread=Cluster, member=1): An existence of a cluster island with senior Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) containing 3 nodes have been detected. Since this Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) is the senior of an older cluster island, the panic protocol is being activated to stop the other island's senior and all junior nodes that belong to it. Member 3 receives the panic: 2010-02-26 15:33:45.803/382.336 Oracle Coherence GE 3.5.3/465p2 <Error> (thread=Cluster, member=3): Received panic from senior Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer) caused by Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer) Member 4, the senior member of the younger cluster, receives the kill message from Member 3: 2010-02-26 15:33:44.921/367.468 Oracle Coherence GE 3.5.3/465p2 <Error> (thread=Cluster, member=4): Received a Kill message from a valid Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer); stopping cluster service. In turn, Member 4 requests the departure of its junior members 5 and 6: 2010-02-26 15:33:44.921/367.468 Oracle Coherence GE 3.5.3/465p2 <Error> (thread=Cluster, member=4): Received a Kill message from a valid Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer); stopping cluster service. 2010-02-26 15:33:43.343/349.015 Oracle Coherence GE 3.5.3/465p2 <Error> (thread=Cluster, member=6): Received a Kill message from a valid Member(Id=4, Timestamp=2010-02-26 15:27:39.574, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer); stopping cluster service. Once Members 4, 5, and 6 restart, they rejoin the original cluster with senior member 1. The log below is from Member 4. Note that it receives a different member id when it rejoins the cluster. 2010-02-26 15:33:44.921/367.468 Oracle Coherence GE 3.5.3/465p2 <Error> (thread=Cluster, member=4): Received a Kill message from a valid Member(Id=3, Timestamp=2010-02-26 15:27:24.892, Address=10.149.155.236:8088, MachineId=1260, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:32459, Role=CoherenceServer); stopping cluster service. 2010-02-26 15:33:46.921/369.468 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Service Cluster left the cluster 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Invocation:InvocationService, member=4): Service InvocationService left the cluster 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=OptimisticCache, member=4): Service OptimisticCache left the cluster 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=ReplicatedCache, member=4): Service ReplicatedCache left the cluster 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=DistributedCache, member=4): Service DistributedCache left the cluster 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Invocation:Management, member=4): Service Management left the cluster 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Member 6 left service Management with senior member 5 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Member 6 left service DistributedCache with senior member 5 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Member 6 left service ReplicatedCache with senior member 5 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Member 6 left service OptimisticCache with senior member 5 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Member 6 left service InvocationService with senior member 5 2010-02-26 15:33:47.046/369.593 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=4): Member(Id=6, Timestamp=2010-02-26 15:33:47.046, Address=10.149.155.78:8088, MachineId=1102, Location=process:228, Role=CoherenceServer) left Cluster with senior member 4 2010-02-26 15:33:49.218/371.765 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=main, member=n/a): Restarting cluster 2010-02-26 15:33:49.421/371.968 Oracle Coherence GE 3.5.3/465p2 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a 2010-02-26 15:33:49.625/372.172 Oracle Coherence GE 3.5.3/465p2 <Info> (thread=Cluster, member=n/a): This Member(Id=5, Timestamp=2010-02-26 15:33:50.499, Address=10.149.155.75:8088, MachineId=1099, Location=process:800, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) joined cluster "cluster:0xDDEB" with senior Member(Id=1, Timestamp=2010-02-26 15:27:06.931, Address=10.149.155.76:8088, MachineId=1100, Location=site:usdhcp.oraclecorp.com,machine:dhcp-burlington6-4fl-east-10-149,process:511, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) Cool isn't it?

Read the article

Improving Partitioned Table Join Performance

- by Paul White

The query optimizer does not always choose an optimal strategy when joining partitioned tables. This post looks at an example, showing how a manual rewrite of the query can almost double performance, while reducing the memory grant to almost nothing. Test Data The two tables in this example use a common partitioning partition scheme. The partition function uses 41 equal-size partitions: CREATE PARTITION FUNCTION PFT (integer) AS RANGE RIGHT FOR VALUES ( 125000, 250000, 375000, 500000, 625000, 750000, 875000, 1000000, 1125000, 1250000, 1375000, 1500000, 1625000, 1750000, 1875000, 2000000, 2125000, 2250000, 2375000, 2500000, 2625000, 2750000, 2875000, 3000000, 3125000, 3250000, 3375000, 3500000, 3625000, 3750000, 3875000, 4000000, 4125000, 4250000, 4375000, 4500000, 4625000, 4750000, 4875000, 5000000 ); GO CREATE PARTITION SCHEME PST AS PARTITION PFT ALL TO ([PRIMARY]); There two tables are: CREATE TABLE dbo.T1 ( TID integer NOT NULL IDENTITY(0,1), Column1 integer NOT NULL, Padding binary(100) NOT NULL DEFAULT 0x, CONSTRAINT PK_T1 PRIMARY KEY CLUSTERED (TID) ON PST (TID) ); CREATE TABLE dbo.T2 ( TID integer NOT NULL, Column1 integer NOT NULL, Padding binary(100) NOT NULL DEFAULT 0x, CONSTRAINT PK_T2 PRIMARY KEY CLUSTERED (TID, Column1) ON PST (TID) ); The next script loads 5 million rows into T1 with a pseudo-random value between 1 and 5 for Column1. The table is partitioned on the IDENTITY column TID: INSERT dbo.T1 WITH (TABLOCKX) (Column1) SELECT (ABS(CHECKSUM(NEWID())) % 5) + 1 FROM dbo.Numbers AS N WHERE n BETWEEN 1 AND 5000000; In case you don’t already have an auxiliary table of numbers lying around, here’s a script to create one with 10 million rows: CREATE TABLE dbo.Numbers (n bigint PRIMARY KEY); WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1), L1 AS(SELECT 1 AS c FROM L0 AS A CROSS JOIN L0 AS B), L2 AS(SELECT 1 AS c FROM L1 AS A CROSS JOIN L1 AS B), L3 AS(SELECT 1 AS c FROM L2 AS A CROSS JOIN L2 AS B), L4 AS(SELECT 1 AS c FROM L3 AS A CROSS JOIN L3 AS B), L5 AS(SELECT 1 AS c FROM L4 AS A CROSS JOIN L4 AS B), Nums AS(SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS n FROM L5) INSERT dbo.Numbers WITH (TABLOCKX) SELECT TOP (10000000) n FROM Nums ORDER BY n OPTION (MAXDOP 1); Table T1 contains data like this: Next we load data into table T2. The relationship between the two tables is that table 2 contains ‘n’ rows for each row in table 1, where ‘n’ is determined by the value in Column1 of table T1. There is nothing particularly special about the data or distribution, by the way. INSERT dbo.T2 WITH (TABLOCKX) (TID, Column1) SELECT T.TID, N.n FROM dbo.T1 AS T JOIN dbo.Numbers AS N ON N.n >= 1 AND N.n <= T.Column1; Table T2 ends up containing about 15 million rows: The primary key for table T2 is a combination of TID and Column1. The data is partitioned according to the value in column TID alone. Partition Distribution The following query shows the number of rows in each partition of table T1: SELECT PartitionID = CA1.P, NumRows = COUNT_BIG(*) FROM dbo.T1 AS T CROSS APPLY (VALUES ($PARTITION.PFT(TID))) AS CA1 (P) GROUP BY CA1.P ORDER BY CA1.P; There are 40 partitions containing 125,000 rows (40 * 125k = 5m rows). The rightmost partition remains empty. The next query shows the distribution for table 2: SELECT PartitionID = CA1.P, NumRows = COUNT_BIG(*) FROM dbo.T2 AS T CROSS APPLY (VALUES ($PARTITION.PFT(TID))) AS CA1 (P) GROUP BY CA1.P ORDER BY CA1.P; There are roughly 375,000 rows in each partition (the rightmost partition is also empty): Ok, that’s the test data done. Test Query and Execution Plan The task is to count the rows resulting from joining tables 1 and 2 on the TID column: SET STATISTICS IO ON; DECLARE @s datetime2 = SYSUTCDATETIME(); SELECT COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID; SELECT DATEDIFF(Millisecond, @s, SYSUTCDATETIME()); SET STATISTICS IO OFF; The optimizer chooses a plan using parallel hash join, and partial aggregation: The Plan Explorer plan tree view shows accurate cardinality estimates and an even distribution of rows across threads (click to enlarge the image): With a warm data cache, the STATISTICS IO output shows that no physical I/O was needed, and all 41 partitions were touched: Running the query without actual execution plan or STATISTICS IO information for maximum performance, the query returns in around 2600ms. Execution Plan Analysis The first step toward improving on the execution plan produced by the query optimizer is to understand how it works, at least in outline. The two parallel Clustered Index Scans use multiple threads to read rows from tables T1 and T2. Parallel scan uses a demand-based scheme where threads are given page(s) to scan from the table as needed. This arrangement has certain important advantages, but does result in an unpredictable distribution of rows amongst threads. The point is that multiple threads cooperate to scan the whole table, but it is impossible to predict which rows end up on which threads. For correct results from the parallel hash join, the execution plan has to ensure that rows from T1 and T2 that might join are processed on the same thread. For example, if a row from T1 with join key value ‘1234’ is placed in thread 5’s hash table, the execution plan must guarantee that any rows from T2 that also have join key value ‘1234’ probe thread 5’s hash table for matches. The way this guarantee is enforced in this parallel hash join plan is by repartitioning rows to threads after each parallel scan. The two repartitioning exchanges route rows to threads using a hash function over the hash join keys. The two repartitioning exchanges use the same hash function so rows from T1 and T2 with the same join key must end up on the same hash join thread. Expensive Exchanges This business of repartitioning rows between threads can be very expensive, especially if a large number of rows is involved. The execution plan selected by the optimizer moves 5 million rows through one repartitioning exchange and around 15 million across the other. As a first step toward removing these exchanges, consider the execution plan selected by the optimizer if we join just one partition from each table, disallowing parallelism: SELECT COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID WHERE $PARTITION.PFT(T1.TID) = 1 AND $PARTITION.PFT(T2.TID) = 1 OPTION (MAXDOP 1); The optimizer has chosen a (one-to-many) merge join instead of a hash join. The single-partition query completes in around 100ms. If everything scaled linearly, we would expect that extending this strategy to all 40 populated partitions would result in an execution time around 4000ms. Using parallelism could reduce that further, perhaps to be competitive with the parallel hash join chosen by the optimizer. This raises a question. If the most efficient way to join one partition from each of the tables is to use a merge join, why does the optimizer not choose a merge join for the full query? Forcing a Merge Join Let’s force the optimizer to use a merge join on the test query using a hint: SELECT COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID OPTION (MERGE JOIN); This is the execution plan selected by the optimizer: This plan results in the same number of logical reads reported previously, but instead of 2600ms the query takes 5000ms. The natural explanation for this drop in performance is that the merge join plan is only using a single thread, whereas the parallel hash join plan could use multiple threads. Parallel Merge Join We can get a parallel merge join plan using the same query hint as before, and adding trace flag 8649: SELECT COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID OPTION (MERGE JOIN, QUERYTRACEON 8649); The execution plan is: This looks promising. It uses a similar strategy to distribute work across threads as seen for the parallel hash join. In practice though, performance is disappointing. On a typical run, the parallel merge plan runs for around 8400ms; slower than the single-threaded merge join plan (5000ms) and much worse than the 2600ms for the parallel hash join. We seem to be going backwards! The logical reads for the parallel merge are still exactly the same as before, with no physical IOs. The cardinality estimates and thread distribution are also still very good (click to enlarge): A big clue to the reason for the poor performance is shown in the wait statistics (captured by Plan Explorer Pro): CXPACKET waits require careful interpretation, and are most often benign, but in this case excessive waiting occurs at the repartitioning exchanges. Unlike the parallel hash join, the repartitioning exchanges in this plan are order-preserving ‘merging’ exchanges (because merge join requires ordered inputs): Parallelism works best when threads can just grab any available unit of work and get on with processing it. Preserving order introduces inter-thread dependencies that can easily lead to significant waits occurring. In extreme cases, these dependencies can result in an intra-query deadlock, though the details of that will have to wait for another time to explore in detail. The potential for waits and deadlocks leads the query optimizer to cost parallel merge join relatively highly, especially as the degree of parallelism (DOP) increases. This high costing resulted in the optimizer choosing a serial merge join rather than parallel in this case. The test results certainly confirm its reasoning. Collocated Joins In SQL Server 2008 and later, the optimizer has another available strategy when joining tables that share a common partition scheme. This strategy is a collocated join, also known as as a per-partition join. It can be applied in both serial and parallel execution plans, though it is limited to 2-way joins in the current optimizer. Whether the optimizer chooses a collocated join or not depends on cost estimation. The primary benefits of a collocated join are that it eliminates an exchange and requires less memory, as we will see next. Costing and Plan Selection The query optimizer did consider a collocated join for our original query, but it was rejected on cost grounds. The parallel hash join with repartitioning exchanges appeared to be a cheaper option. There is no query hint to force a collocated join, so we have to mess with the costing framework to produce one for our test query. Pretending that IOs cost 50 times more than usual is enough to convince the optimizer to use collocated join with our test query: -- Pretend IOs are 50x cost temporarily DBCC SETIOWEIGHT(50); -- Co-located hash join SELECT COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID OPTION (RECOMPILE); -- Reset IO costing DBCC SETIOWEIGHT(1); Collocated Join Plan The estimated execution plan for the collocated join is: The Constant Scan contains one row for each partition of the shared partitioning scheme, from 1 to 41. The hash repartitioning exchanges seen previously are replaced by a single Distribute Streams exchange using Demand partitioning. Demand partitioning means that the next partition id is given to the next parallel thread that asks for one. My test machine has eight logical processors, and all are available for SQL Server to use. As a result, there are eight threads in the single parallel branch in this plan, each processing one partition from each table at a time. Once a thread finishes processing a partition, it grabs a new partition number from the Distribute Streams exchange…and so on until all partitions have been processed. It is important to understand that the parallel scans in this plan are different from the parallel hash join plan. Although the scans have the same parallelism icon, tables T1 and T2 are not being co-operatively scanned by multiple threads in the same way. Each thread reads a single partition of T1 and performs a hash match join with the same partition from table T2. The properties of the two Clustered Index Scans show a Seek Predicate (unusual for a scan!) limiting the rows to a single partition: The crucial point is that the join between T1 and T2 is on TID, and TID is the partitioning column for both tables. A thread that processes partition ‘n’ is guaranteed to see all rows that can possibly join on TID for that partition. In addition, no other thread will see rows from that partition, so this removes the need for repartitioning exchanges. CPU and Memory Efficiency Improvements The collocated join has removed two expensive repartitioning exchanges and added a single exchange processing 41 rows (one for each partition id). Remember, the parallel hash join plan exchanges had to process 5 million and 15 million rows. The amount of processor time spent on exchanges will be much lower in the collocated join plan. In addition, the collocated join plan has a maximum of 8 threads processing single partitions at any one time. The 41 partitions will all be processed eventually, but a new partition is not started until a thread asks for it. Threads can reuse hash table memory for the new partition. The parallel hash join plan also had 8 hash tables, but with all 5,000,000 build rows loaded at the same time. The collocated plan needs memory for only 8 * 125,000 = 1,000,000 rows at any one time. Collocated Hash Join Performance The collated join plan has disappointing performance in this case. The query runs for around 25,300ms despite the same IO statistics as usual. This is much the worst result so far, so what went wrong? It turns out that cardinality estimation for the single partition scans of table T1 is slightly low. The properties of the Clustered Index Scan of T1 (graphic immediately above) show the estimation was for 121,951 rows. This is a small shortfall compared with the 125,000 rows actually encountered, but it was enough to cause the hash join to spill to physical tempdb: A level 1 spill doesn’t sound too bad, until you realize that the spill to tempdb probably occurs for each of the 41 partitions. As a side note, the cardinality estimation error is a little surprising because the system tables accurately show there are 125,000 rows in every partition of T1. Unfortunately, the optimizer uses regular column and index statistics to derive cardinality estimates here rather than system table information (e.g. sys.partitions). Collocated Merge Join We will never know how well the collocated parallel hash join plan might have worked without the cardinality estimation error (and the resulting 41 spills to tempdb) but we do know: Merge join does not require a memory grant; and Merge join was the optimizer’s preferred join option for a single partition join Putting this all together, what we would really like to see is the same collocated join strategy, but using merge join instead of hash join. Unfortunately, the current query optimizer cannot produce a collocated merge join; it only knows how to do collocated hash join. So where does this leave us? CROSS APPLY sys.partitions We can try to write our own collocated join query. We can use sys.partitions to find the partition numbers, and CROSS APPLY to get a count per partition, with a final step to sum the partial counts. The following query implements this idea: SELECT row_count = SUM(Subtotals.cnt) FROM ( -- Partition numbers SELECT p.partition_number FROM sys.partitions AS p WHERE p.[object_id] = OBJECT_ID(N'T1', N'U') AND p.index_id = 1 ) AS P CROSS APPLY ( -- Count per collocated join SELECT cnt = COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID WHERE $PARTITION.PFT(T1.TID) = p.partition_number AND $PARTITION.PFT(T2.TID) = p.partition_number ) AS SubTotals; The estimated plan is: The cardinality estimates aren’t all that good here, especially the estimate for the scan of the system table underlying the sys.partitions view. Nevertheless, the plan shape is heading toward where we would like to be. Each partition number from the system table results in a per-partition scan of T1 and T2, a one-to-many Merge Join, and a Stream Aggregate to compute the partial counts. The final Stream Aggregate just sums the partial counts. Execution time for this query is around 3,500ms, with the same IO statistics as always. This compares favourably with 5,000ms for the serial plan produced by the optimizer with the OPTION (MERGE JOIN) hint. This is another case of the sum of the parts being less than the whole – summing 41 partial counts from 41 single-partition merge joins is faster than a single merge join and count over all partitions. Even so, this single-threaded collocated merge join is not as quick as the original parallel hash join plan, which executed in 2,600ms. On the positive side, our collocated merge join uses only one logical processor and requires no memory grant. The parallel hash join plan used 16 threads and reserved 569 MB of memory: Using a Temporary Table Our collocated merge join plan should benefit from parallelism. The reason parallelism is not being used is that the query references a system table. We can work around that by writing the partition numbers to a temporary table (or table variable): SET STATISTICS IO ON; DECLARE @s datetime2 = SYSUTCDATETIME(); CREATE TABLE #P ( partition_number integer PRIMARY KEY); INSERT #P (partition_number) SELECT p.partition_number FROM sys.partitions AS p WHERE p.[object_id] = OBJECT_ID(N'T1', N'U') AND p.index_id = 1; SELECT row_count = SUM(Subtotals.cnt) FROM #P AS p CROSS APPLY ( SELECT cnt = COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID WHERE $PARTITION.PFT(T1.TID) = p.partition_number AND $PARTITION.PFT(T2.TID) = p.partition_number ) AS SubTotals; DROP TABLE #P; SELECT DATEDIFF(Millisecond, @s, SYSUTCDATETIME()); SET STATISTICS IO OFF; Using the temporary table adds a few logical reads, but the overall execution time is still around 3500ms, indistinguishable from the same query without the temporary table. The problem is that the query optimizer still doesn’t choose a parallel plan for this query, though the removal of the system table reference means that it could if it chose to: In fact the optimizer did enter the parallel plan phase of query optimization (running search 1 for a second time): Unfortunately, the parallel plan found seemed to be more expensive than the serial plan. This is a crazy result, caused by the optimizer’s cost model not reducing operator CPU costs on the inner side of a nested loops join. Don’t get me started on that, we’ll be here all night. In this plan, everything expensive happens on the inner side of a nested loops join. Without a CPU cost reduction to compensate for the added cost of exchange operators, candidate parallel plans always look more expensive to the optimizer than the equivalent serial plan. Parallel Collocated Merge Join We can produce the desired parallel plan using trace flag 8649 again: SELECT row_count = SUM(Subtotals.cnt) FROM #P AS p CROSS APPLY ( SELECT cnt = COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID WHERE $PARTITION.PFT(T1.TID) = p.partition_number AND $PARTITION.PFT(T2.TID) = p.partition_number ) AS SubTotals OPTION (QUERYTRACEON 8649); The actual execution plan is: One difference between this plan and the collocated hash join plan is that a Repartition Streams exchange operator is used instead of Distribute Streams. The effect is similar, though not quite identical. The Repartition uses round-robin partitioning, meaning the next partition id is pushed to the next thread in sequence. The Distribute Streams exchange seen earlier used Demand partitioning, meaning the next partition id is pulled across the exchange by the next thread that is ready for more work. There are subtle performance implications for each partitioning option, but going into that would again take us too far off the main point of this post. Performance The important thing is the performance of this parallel collocated merge join – just 1350ms on a typical run. The list below shows all the alternatives from this post (all timings include creation, population, and deletion of the temporary table where appropriate) from quickest to slowest: Collocated parallel merge join: 1350ms Parallel hash join: 2600ms Collocated serial merge join: 3500ms Serial merge join: 5000ms Parallel merge join: 8400ms Collated parallel hash join: 25,300ms (hash spill per partition) The parallel collocated merge join requires no memory grant (aside from a paltry 1.2MB used for exchange buffers). This plan uses 16 threads at DOP 8; but 8 of those are (rather pointlessly) allocated to the parallel scan of the temporary table. These are minor concerns, but it turns out there is a way to address them if it bothers you. Parallel Collocated Merge Join with Demand Partitioning This final tweak replaces the temporary table with a hard-coded list of partition ids (dynamic SQL could be used to generate this query from sys.partitions): SELECT row_count = SUM(Subtotals.cnt) FROM ( VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (11),(12),(13),(14),(15),(16),(17),(18),(19),(20), (21),(22),(23),(24),(25),(26),(27),(28),(29),(30), (31),(32),(33),(34),(35),(36),(37),(38),(39),(40),(41) ) AS P (partition_number) CROSS APPLY ( SELECT cnt = COUNT_BIG(*) FROM dbo.T1 AS T1 JOIN dbo.T2 AS T2 ON T2.TID = T1.TID WHERE $PARTITION.PFT(T1.TID) = p.partition_number AND $PARTITION.PFT(T2.TID) = p.partition_number ) AS SubTotals OPTION (QUERYTRACEON 8649); The actual execution plan is: The parallel collocated hash join plan is reproduced below for comparison: The manual rewrite has another advantage that has not been mentioned so far: the partial counts (per partition) can be computed earlier than the partial counts (per thread) in the optimizer’s collocated join plan. The earlier aggregation is performed by the extra Stream Aggregate under the nested loops join. The performance of the parallel collocated merge join is unchanged at around 1350ms. Final Words It is a shame that the current query optimizer does not consider a collocated merge join (Connect item closed as Won’t Fix). The example used in this post showed an improvement in execution time from 2600ms to 1350ms using a modestly-sized data set and limited parallelism. In addition, the memory requirement for the query was almost completely eliminated – down from 569MB to 1.2MB. The problem with the parallel hash join selected by the optimizer is that it attempts to process the full data set all at once (albeit using eight threads). It requires a large memory grant to hold all 5 million rows from table T1 across the eight hash tables, and does not take advantage of the divide-and-conquer opportunity offered by the common partitioning. The great thing about the collocated join strategies is that each parallel thread works on a single partition from both tables, reading rows, performing the join, and computing a per-partition subtotal, before moving on to a new partition. From a thread’s point of view… If you have trouble visualizing what is happening from just looking at the parallel collocated merge join execution plan, let’s look at it again, but from the point of view of just one thread operating between the two Parallelism (exchange) operators. Our thread picks up a single partition id from the Distribute Streams exchange, and starts a merge join using ordered rows from partition 1 of table T1 and partition 1 of table T2. By definition, this is all happening on a single thread. As rows join, they are added to a (per-partition) count in the Stream Aggregate immediately above the Merge Join. Eventually, either T1 (partition 1) or T2 (partition 1) runs out of rows and the merge join stops. The per-partition count from the aggregate passes on through the Nested Loops join to another Stream Aggregate, which is maintaining a per-thread subtotal. Our same thread now picks up a new partition id from the exchange (say it gets id 9 this time). The count in the per-partition aggregate is reset to zero, and the processing of partition 9 of both tables proceeds just as it did for partition 1, and on the same thread. Each thread picks up a single partition id and processes all the data for that partition, completely independently from other threads working on other partitions. One thread might eventually process partitions (1, 9, 17, 25, 33, 41) while another is concurrently processing partitions (2, 10, 18, 26, 34) and so on for the other six threads at DOP 8. The point is that all 8 threads can execute independently and concurrently, continuing to process new partitions until the wider job (of which the thread has no knowledge!) is done. This divide-and-conquer technique can be much more efficient than simply splitting the entire workload across eight threads all at once. Related Reading Understanding and Using Parallelism in SQL Server Parallel Execution Plans Suck © 2013 Paul White – All Rights Reserved Twitter: @SQL_Kiwi

Read the article

Advanced TSQL Tuning: Why Internals Knowledge Matters

- by Paul White

There is much more to query tuning than reducing logical reads and adding covering nonclustered indexes. Query tuning is not complete as soon as the query returns results quickly in the development or test environments. In production, your query will compete for memory, CPU, locks, I/O and other resources on the server. Today’s entry looks at some tuning considerations that are often overlooked, and shows how deep internals knowledge can help you write better TSQL. As always, we’ll need some example data. In fact, we are going to use three tables today, each of which is structured like this: Each table has 50,000 rows made up of an INTEGER id column and a padding column containing 3,999 characters in every row. The only difference between the three tables is in the type of the padding column: the first table uses CHAR(3999), the second uses VARCHAR(MAX), and the third uses the deprecated TEXT type. A script to create a database with the three tables and load the sample data follows: USE master; GO IF DB_ID('SortTest') IS NOT NULL DROP DATABASE SortTest; GO CREATE DATABASE SortTest COLLATE LATIN1_GENERAL_BIN; GO ALTER DATABASE SortTest MODIFY FILE ( NAME = 'SortTest', SIZE = 3GB, MAXSIZE = 3GB ); GO ALTER DATABASE SortTest MODIFY FILE ( NAME = 'SortTest_log', SIZE = 256MB, MAXSIZE = 1GB, FILEGROWTH = 128MB ); GO ALTER DATABASE SortTest SET ALLOW_SNAPSHOT_ISOLATION OFF ; ALTER DATABASE SortTest SET AUTO_CLOSE OFF ; ALTER DATABASE SortTest SET AUTO_CREATE_STATISTICS ON ; ALTER DATABASE SortTest SET AUTO_SHRINK OFF ; ALTER DATABASE SortTest SET AUTO_UPDATE_STATISTICS ON ; ALTER DATABASE SortTest SET AUTO_UPDATE_STATISTICS_ASYNC ON ; ALTER DATABASE SortTest SET PARAMETERIZATION SIMPLE ; ALTER DATABASE SortTest SET READ_COMMITTED_SNAPSHOT OFF ; ALTER DATABASE SortTest SET MULTI_USER ; ALTER DATABASE SortTest SET RECOVERY SIMPLE ; USE SortTest; GO CREATE TABLE dbo.TestCHAR ( id INTEGER IDENTITY (1,1) NOT NULL, padding CHAR(3999) NOT NULL, CONSTRAINT [PK dbo.TestCHAR (id)] PRIMARY KEY CLUSTERED (id), ) ; CREATE TABLE dbo.TestMAX ( id INTEGER IDENTITY (1,1) NOT NULL, padding VARCHAR(MAX) NOT NULL, CONSTRAINT [PK dbo.TestMAX (id)] PRIMARY KEY CLUSTERED (id), ) ; CREATE TABLE dbo.TestTEXT ( id INTEGER IDENTITY (1,1) NOT NULL, padding TEXT NOT NULL, CONSTRAINT [PK dbo.TestTEXT (id)] PRIMARY KEY CLUSTERED (id), ) ; -- ============= -- Load TestCHAR (about 3s) -- ============= INSERT INTO dbo.TestCHAR WITH (TABLOCKX) ( padding ) SELECT padding = REPLICATE(CHAR(65 + (Data.n % 26)), 3999) FROM ( SELECT TOP (50000) n = ROW_NUMBER() OVER (ORDER BY (SELECT 0)) - 1 FROM master.sys.columns C1, master.sys.columns C2, master.sys.columns C3 ORDER BY n ASC ) AS Data ORDER BY Data.n ASC ; -- ============ -- Load TestMAX (about 3s) -- ============ INSERT INTO dbo.TestMAX WITH (TABLOCKX) ( padding ) SELECT CONVERT(VARCHAR(MAX), padding) FROM dbo.TestCHAR ORDER BY id ; -- ============= -- Load TestTEXT (about 5s) -- ============= INSERT INTO dbo.TestTEXT WITH (TABLOCKX) ( padding ) SELECT CONVERT(TEXT, padding) FROM dbo.TestCHAR ORDER BY id ; -- ========== -- Space used -- ========== -- EXECUTE sys.sp_spaceused @objname = 'dbo.TestCHAR'; EXECUTE sys.sp_spaceused @objname = 'dbo.TestMAX'; EXECUTE sys.sp_spaceused @objname = 'dbo.TestTEXT'; ; CHECKPOINT ; That takes around 15 seconds to run, and shows the space allocated to each table in its output: To illustrate the points I want to make today, the example task we are going to set ourselves is to return a random set of 150 rows from each table. The basic shape of the test query is the same for each of the three test tables: SELECT TOP (150) T.id, T.padding FROM dbo.Test AS T ORDER BY NEWID() OPTION (MAXDOP 1) ; Test 1 – CHAR(3999) Running the template query shown above using the TestCHAR table as the target, we find that the query takes around 5 seconds to return its results. This seems slow, considering that the table only has 50,000 rows. Working on the assumption that generating a GUID for each row is a CPU-intensive operation, we might try enabling parallelism to see if that speeds up the response time. Running the query again (but without the MAXDOP 1 hint) on a machine with eight logical processors, the query now takes 10 seconds to execute – twice as long as when run serially. Rather than attempting further guesses at the cause of the slowness, let’s go back to serial execution and add some monitoring. The script below monitors STATISTICS IO output and the amount of tempdb used by the test query. We will also run a Profiler trace to capture any warnings generated during query execution. DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) TC.id, TC.padding FROM dbo.TestCHAR AS TC ORDER BY NEWID() OPTION (MAXDOP 1) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; Let’s take a closer look at the statistics and query plan generated from this: Following the flow of the data from right to left, we see the expected 50,000 rows emerging from the Clustered Index Scan, with a total estimated size of around 191MB. The Compute Scalar adds a column containing a random GUID (generated from the NEWID() function call) for each row. With this extra column in place, the size of the data arriving at the Sort operator is estimated to be 192MB. Sort is a blocking operator – it has to examine all of the rows on its input before it can produce its first row of output (the last row received might sort first). This characteristic means that Sort requires a memory grant – memory allocated for the query’s use by SQL Server just before execution starts. In this case, the Sort is the only memory-consuming operator in the plan, so it has access to the full 243MB (248,696KB) of memory reserved by SQL Server for this query execution. Notice that the memory grant is significantly larger than the expected size of the data to be sorted. SQL Server uses a number of techniques to speed up sorting, some of which sacrifice size for comparison speed. Sorts typically require a very large number of comparisons, so this is usually a very effective optimization. One of the drawbacks is that it is not possible to exactly predict the sort space needed, as it depends on the data itself. SQL Server takes an educated guess based on data types, sizes, and the number of rows expected, but the algorithm is not perfect. In spite of the large memory grant, the Profiler trace shows a Sort Warning event (indicating that the sort ran out of memory), and the tempdb usage monitor shows that 195MB of tempdb space was used – all of that for system use. The 195MB represents physical write activity on tempdb, because SQL Server strictly enforces memory grants – a query cannot ‘cheat’ and effectively gain extra memory by spilling to tempdb pages that reside in memory. Anyway, the key point here is that it takes a while to write 195MB to disk, and this is the main reason that the query takes 5 seconds overall. If you are wondering why using parallelism made the problem worse, consider that eight threads of execution result in eight concurrent partial sorts, each receiving one eighth of the memory grant. The eight sorts all spilled to tempdb, resulting in inefficiencies as the spilled sorts competed for disk resources. More importantly, there are specific problems at the point where the eight partial results are combined, but I’ll cover that in a future post. CHAR(3999) Performance Summary: 5 seconds elapsed time 243MB memory grant 195MB tempdb usage 192MB estimated sort set 25,043 logical reads Sort Warning Test 2 – VARCHAR(MAX) We’ll now run exactly the same test (with the additional monitoring) on the table using a VARCHAR(MAX) padding column: DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) TM.id, TM.padding FROM dbo.TestMAX AS TM ORDER BY NEWID() OPTION (MAXDOP 1) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; This time the query takes around 8 seconds to complete (3 seconds longer than Test 1). Notice that the estimated row and data sizes are very slightly larger, and the overall memory grant has also increased very slightly to 245MB. The most marked difference is in the amount of tempdb space used – this query wrote almost 391MB of sort run data to the physical tempdb file. Don’t draw any general conclusions about VARCHAR(MAX) versus CHAR from this – I chose the length of the data specifically to expose this edge case. In most cases, VARCHAR(MAX) performs very similarly to CHAR – I just wanted to make test 2 a bit more exciting. MAX Performance Summary: 8 seconds elapsed time 245MB memory grant 391MB tempdb usage 193MB estimated sort set 25,043 logical reads Sort warning Test 3 – TEXT The same test again, but using the deprecated TEXT data type for the padding column: DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) TT.id, TT.padding FROM dbo.TestTEXT AS TT ORDER BY NEWID() OPTION (MAXDOP 1, RECOMPILE) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; This time the query runs in 500ms. If you look at the metrics we have been checking so far, it’s not hard to understand why: TEXT Performance Summary: 0.5 seconds elapsed time 9MB memory grant 5MB tempdb usage 5MB estimated sort set 207 logical reads 596 LOB logical reads Sort warning SQL Server’s memory grant algorithm still underestimates the memory needed to perform the sorting operation, but the size of the data to sort is so much smaller (5MB versus 193MB previously) that the spilled sort doesn’t matter very much. Why is the data size so much smaller? The query still produces the correct results – including the large amount of data held in the padding column – so what magic is being performed here? TEXT versus MAX Storage The answer lies in how columns of the TEXT data type are stored. By default, TEXT data is stored off-row in separate LOB pages – which explains why this is the first query we have seen that records LOB logical reads in its STATISTICS IO output. You may recall from my last post that LOB data leaves an in-row pointer to the separate storage structure holding the LOB data. SQL Server can see that the full LOB value is not required by the query plan until results are returned, so instead of passing the full LOB value down the plan from the Clustered Index Scan, it passes the small in-row structure instead. SQL Server estimates that each row coming from the scan will be 79 bytes long – 11 bytes for row overhead, 4 bytes for the integer id column, and 64 bytes for the LOB pointer (in fact the pointer is rather smaller – usually 16 bytes – but the details of that don’t really matter right now). OK, so this query is much more efficient because it is sorting a very much smaller data set – SQL Server delays retrieving the LOB data itself until after the Sort starts producing its 150 rows. The question that normally arises at this point is: Why doesn’t SQL Server use the same trick when the padding column is defined as VARCHAR(MAX)? The answer is connected with the fact that if the actual size of the VARCHAR(MAX) data is 8000 bytes or less, it is usually stored in-row in exactly the same way as for a VARCHAR(8000) column – MAX data only moves off-row into LOB storage when it exceeds 8000 bytes. The default behaviour of the TEXT type is to be stored off-row by default, unless the ‘text in row’ table option is set suitably and there is room on the page. There is an analogous (but opposite) setting to control the storage of MAX data – the ‘large value types out of row’ table option. By enabling this option for a table, MAX data will be stored off-row (in a LOB structure) instead of in-row. SQL Server Books Online has good coverage of both options in the topic In Row Data. The MAXOOR Table The essential difference, then, is that MAX defaults to in-row storage, and TEXT defaults to off-row (LOB) storage. You might be thinking that we could get the same benefits seen for the TEXT data type by storing the VARCHAR(MAX) values off row – so let’s look at that option now. This script creates a fourth table, with the VARCHAR(MAX) data stored off-row in LOB pages: CREATE TABLE dbo.TestMAXOOR ( id INTEGER IDENTITY (1,1) NOT NULL, padding VARCHAR(MAX) NOT NULL, CONSTRAINT [PK dbo.TestMAXOOR (id)] PRIMARY KEY CLUSTERED (id), ) ; EXECUTE sys.sp_tableoption @TableNamePattern = N'dbo.TestMAXOOR', @OptionName = 'large value types out of row', @OptionValue = 'true' ; SELECT large_value_types_out_of_row FROM sys.tables WHERE [schema_id] = SCHEMA_ID(N'dbo') AND name = N'TestMAXOOR' ; INSERT INTO dbo.TestMAXOOR WITH (TABLOCKX) ( padding ) SELECT SPACE(0) FROM dbo.TestCHAR ORDER BY id ; UPDATE TM WITH (TABLOCK) SET padding.WRITE (TC.padding, NULL, NULL) FROM dbo.TestMAXOOR AS TM JOIN dbo.TestCHAR AS TC ON TC.id = TM.id ; EXECUTE sys.sp_spaceused @objname = 'dbo.TestMAXOOR' ; CHECKPOINT ; Test 4 – MAXOOR We can now re-run our test on the MAXOOR (MAX out of row) table: DECLARE @read BIGINT, @write BIGINT ; SELECT @read = SUM(num_of_bytes_read), @write = SUM(num_of_bytes_written) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; SET STATISTICS IO ON ; SELECT TOP (150) MO.id, MO.padding FROM dbo.TestMAXOOR AS MO ORDER BY NEWID() OPTION (MAXDOP 1, RECOMPILE) ; SET STATISTICS IO OFF ; SELECT tempdb_read_MB = (SUM(num_of_bytes_read) - @read) / 1024. / 1024., tempdb_write_MB = (SUM(num_of_bytes_written) - @write) / 1024. / 1024., internal_use_MB = ( SELECT internal_objects_alloc_page_count / 128.0 FROM sys.dm_db_task_space_usage WHERE session_id = @@SPID ) FROM tempdb.sys.database_files AS DBF JOIN sys.dm_io_virtual_file_stats(2, NULL) AS FS ON FS.file_id = DBF.file_id WHERE DBF.type_desc = 'ROWS' ; TEXT Performance Summary: 0.3 seconds elapsed time 245MB memory grant 0MB tempdb usage 193MB estimated sort set 207 logical reads 446 LOB logical reads No sort warning The query runs very quickly – slightly faster than Test 3, and without spilling the sort to tempdb (there is no sort warning in the trace, and the monitoring query shows zero tempdb usage by this query). SQL Server is passing the in-row pointer structure down the plan and only looking up the LOB value on the output side of the sort. The Hidden Problem There is still a huge problem with this query though – it requires a 245MB memory grant. No wonder the sort doesn’t spill to tempdb now – 245MB is about 20 times more memory than this query actually requires to sort 50,000 records containing LOB data pointers. Notice that the estimated row and data sizes in the plan are the same as in test 2 (where the MAX data was stored in-row). The optimizer assumes that MAX data is stored in-row, regardless of the sp_tableoption setting ‘large value types out of row’. Why? Because this option is dynamic – changing it does not immediately force all MAX data in the table in-row or off-row, only when data is added or actually changed. SQL Server does not keep statistics to show how much MAX or TEXT data is currently in-row, and how much is stored in LOB pages. This is an annoying limitation, and one which I hope will be addressed in a future version of the product. So why should we worry about this? Excessive memory grants reduce concurrency and may result in queries waiting on the RESOURCE_SEMAPHORE wait type while they wait for memory they do not need. 245MB is an awful lot of memory, especially on 32-bit versions where memory grants cannot use AWE-mapped memory. Even on a 64-bit server with plenty of memory, do you really want a single query to consume 0.25GB of memory unnecessarily? That’s 32,000 8KB pages that might be put to much better use. The Solution The answer is not to use the TEXT data type for the padding column. That solution happens to have better performance characteristics for this specific query, but it still results in a spilled sort, and it is hard to recommend the use of a data type which is scheduled for removal. I hope it is clear to you that the fundamental problem here is that SQL Server sorts the whole set arriving at a Sort operator. Clearly, it is not efficient to sort the whole table in memory just to return 150 rows in a random order. The TEXT example was more efficient because it dramatically reduced the size of the set that needed to be sorted. We can do the same thing by selecting 150 unique keys from the table at random (sorting by NEWID() for example) and only then retrieving the large padding column values for just the 150 rows we need. The following script implements that idea for all four tables: SET STATISTICS IO ON ; WITH TestTable AS ( SELECT * FROM dbo.TestCHAR ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id = ANY (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; WITH TestTable AS ( SELECT * FROM dbo.TestMAX ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id IN (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; WITH TestTable AS ( SELECT * FROM dbo.TestTEXT ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id IN (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; WITH TestTable AS ( SELECT * FROM dbo.TestMAXOOR ), TopKeys AS ( SELECT TOP (150) id FROM TestTable ORDER BY NEWID() ) SELECT T1.id, T1.padding FROM TestTable AS T1 WHERE T1.id IN (SELECT id FROM TopKeys) OPTION (MAXDOP 1) ; SET STATISTICS IO OFF ; All four queries now return results in much less than a second, with memory grants between 6 and 12MB, and without spilling to tempdb. The small remaining inefficiency is in reading the id column values from the clustered primary key index. As a clustered index, it contains all the in-row data at its leaf. The CHAR and VARCHAR(MAX) tables store the padding column in-row, so id values are separated by a 3999-character column, plus row overhead. The TEXT and MAXOOR tables store the padding values off-row, so id values in the clustered index leaf are separated by the much-smaller off-row pointer structure. This difference is reflected in the number of logical page reads performed by the four queries: Table 'TestCHAR' logical reads 25511 lob logical reads 000 Table 'TestMAX'. logical reads 25511 lob logical reads 000 Table 'TestTEXT' logical reads 00412 lob logical reads 597 Table 'TestMAXOOR' logical reads 00413 lob logical reads 446 We can increase the density of the id values by creating a separate nonclustered index on the id column only. This is the same key as the clustered index, of course, but the nonclustered index will not include the rest of the in-row column data. CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestCHAR (id); CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestMAX (id); CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestTEXT (id); CREATE UNIQUE NONCLUSTERED INDEX uq1 ON dbo.TestMAXOOR (id); The four queries can now use the very dense nonclustered index to quickly scan the id values, sort them by NEWID(), select the 150 ids we want, and then look up the padding data. The logical reads with the new indexes in place are: Table 'TestCHAR' logical reads 835 lob logical reads 0 Table 'TestMAX' logical reads 835 lob logical reads 0 Table 'TestTEXT' logical reads 686 lob logical reads 597 Table 'TestMAXOOR' logical reads 686 lob logical reads 448 With the new index, all four queries use the same query plan (click to enlarge): Performance Summary: 0.3 seconds elapsed time 6MB memory grant 0MB tempdb usage 1MB sort set 835 logical reads (CHAR, MAX) 686 logical reads (TEXT, MAXOOR) 597 LOB logical reads (TEXT) 448 LOB logical reads (MAXOOR) No sort warning I’ll leave it as an exercise for the reader to work out why trying to eliminate the Key Lookup by adding the padding column to the new nonclustered indexes would be a daft idea Conclusion This post is not about tuning queries that access columns containing big strings. It isn’t about the internal differences between TEXT and MAX data types either. It isn’t even about the cool use of UPDATE .WRITE used in the MAXOOR table load. No, this post is about something else: Many developers might not have tuned our starting example query at all – 5 seconds isn’t that bad, and the original query plan looks reasonable at first glance. Perhaps the NEWID() function would have been blamed for ‘just being slow’ – who knows. 5 seconds isn’t awful – unless your users expect sub-second responses – but using 250MB of memory and writing 200MB to tempdb certainly is! If ten sessions ran that query at the same time in production that’s 2.5GB of memory usage and 2GB hitting tempdb. Of course, not all queries can be rewritten to avoid large memory grants and sort spills using the key-lookup technique in this post, but that’s not the point either. The point of this post is that a basic understanding of execution plans is not enough. Tuning for logical reads and adding covering indexes is not enough. If you want to produce high-quality, scalable TSQL that won’t get you paged as soon as it hits production, you need a deep understanding of execution plans, and as much accurate, deep knowledge about SQL Server as you can lay your hands on. The advanced database developer has a wide range of tools to use in writing queries that perform well in a range of circumstances. By the way, the examples in this post were written for SQL Server 2008. They will run on 2005 and demonstrate the same principles, but you won’t get the same figures I did because 2005 had a rather nasty bug in the Top N Sort operator. Fair warning: if you do decide to run the scripts on a 2005 instance (particularly the parallel query) do it before you head out for lunch… This post is dedicated to the people of Christchurch, New Zealand. © 2011 Paul White email: @[email protected] twitter: @SQL_Kiwi

Search Results

Search found 7325 results on 293 pages for 'windows7 32'.

Page 287/293 | < Previous Page | 283 284 285 286 287 288 289 290 291 292 293 | Next Page >

- by Eddie

- by Bernd Haug

- by Tunji Gbadamosi

- by brianheys

- by Joshua Moore

- by ilektron

- by Tim

- by StM

- by user289637

- by DaTeNtImE

- by king.net

- by Jason

- by user308226

- by user663414

- by Ankita Agrawal

- by Dinedal

- by user1602079

- by RealityGone

- by sarcozona

- by TheMorfeus

- by Jason Wells

- by Rick Strahl

- by Ricardo Ferreira

- by Paul White

- by Paul White

< Previous Page | 283 284 285 286 287 288 289 290 291 292 293 | Next Page >