normalize - Page 15 - Developer IT

Logging Application Block

- by Gordon Carpenter-Thompson

I'm using the Logging Application Block in my ASP.NET application and want to convert the application to a Sharepoint WebPart. It all works fine as long as I change: <trust level="WSS_Minimal" originUrl="" /> to <trust level="Full" originUrl="" /> If not I get an exception in the logs: Failed to add webpart *************255Fcatalogs%252Fwp%252FSearchWebPart%252Ewebpart;SearchWebPart. Exception Microsoft.SharePoint.WebPartPages.WebPartPageUserException: The type initializer for 'Microsoft.Practices.EnterpriseLibrary.Logging.Logger' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Practices.EnterpriseLibrary.Logging.Logger' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Practices.EnterpriseLibrary.Common.Configuration.SystemConfigurationSource' threw an exception. ---> System.Security.SecurityException: Request for the permission of type 'System.Security.Permissions.FileIOPermission, mscorlib, Version=2.0.0.0, Culture=neut... ...ral, PublicKeyToken=b77a5c561934e089' failed. at System.Security.CodeAccessSecurityEngine.Check(Object demand, StackCrawlMark& stackMark, Boolean isPermSet) at System.Security.CodeAccessPermission.Demand() at System.AppDomainSetup.VerifyDir(String dir, Boolean normalize) at System.AppDomainSetup.get_ConfigurationFile() at Microsoft.Practices.EnterpriseLibrary.Common.Configuration.SystemConfigurationSourceImplementation..ctor(Boolean refresh) at Microsoft.Practices.EnterpriseLibrary.Common.Configuration.SystemConfigurationSource..cctor() The action that failed was: Demand The type of the first permission that failed was: System.Security.Permissions.FileIOPermission The first permission that failed was: <IPermission class="System.Security.Permissions.FileIOPermi... ...ssion, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" PathDiscovery="D:\Inetpub\wwwroot\wss\VirtualDirectories\8686\web.config"/> The demand was for: <IPermission class="System.Security.Permissions.FileIOPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" PathDiscovery="D:\Inetpub\wwwroot\wss\VirtualDirectories\8686\web.config"/> The granted set of the failing assembly was: <PermissionSet class="System.Security.PermissionSet" version="1"> <IPermission class="System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Flags="Execution"/> <IPermission class="System.Security.Permissions.StrongNameIdentityPermis... ...sion, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" PublicKeyBlob="0024000004800000940000000602000000240000525341310004000001000100257FAE170ABB2AB4A8EF914DFEA757F7DB8C221F01850FC8753A4C6585C0B07749DA33DF4D64A721A070E7CDCDEFC8C786E3626418389BCF461E4300E6F4C477BE5CE64AD12C29D517208D6BA627D9F73A9066B7638BE1FEE3EABE6C3E537B546CB3B5DE5E436F95278BB1E9DBDE85C2A6B624010A8073841D467CC7A0A0C6C8" Name="Microsoft.Practices.EnterpriseLibrary.Common" AssemblyVersion="3.1.0.0"/> <IPermission class="System.Security.Permissions.UrlIdentityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Url="file:///D:/Inetpub/wwwroot/wss/VirtualDirectories/8686/bin/Microsoft.Practices.EnterpriseLibrary.Common.DLL"/> <IPe... ...rmission class="System.Security.Permissions.ZoneIdentityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Zone="MyComputer"/> <IPermission class="System.Web.AspNetHostingPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Level="Minimal"/> <IPermission class="Microsoft.SharePoint.Security.WebPartPermission, Microsoft.SharePoint.Security, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" version="1" Connections="True"/> </PermissionSet> The assembly or AppDomain that failed was: Microsoft.Practices.EnterpriseLibrary.Common, Version=3.1.0.0, Culture=neutral, PublicKeyToken=a646907c4a695009 The Zone of the assembly that failed was: MyComputer The Url of the assem... ...bly that failed was: file:///D:/Inetpub/wwwroot/wss/VirtualDirectories/8686/bin/Microsoft.Practices.EnterpriseLibrary.Common.DLL --- End of inner exception stack trace --- at Microsoft.Practices.EnterpriseLibrary.Common.Configuration.SystemConfigurationSource..ctor() at Microsoft.Practices.EnterpriseLibrary.Common.Configuration.ConfigurationSourceFactory.Create() at Microsoft.Practices.EnterpriseLibrary.Logging.Logger..cctor() --- End of inner exception stack trace --- at Microsoft.Practices.EnterpriseLibrary.Logging.Logger.Write(LogEntry log) at com.okana.sharepoint.SearchWebPart.OnLoad(EventArgs e) at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Control.AddedControl(Control control, Int32 index) at System.Web.UI.ControlCollection.Add(Con... ...trol child) at System.Web.UI.WebControls.WebParts.WebPartManager.WebPartManagerControlCollection.AddWebPartHelper(WebPart webPart) at System.Web.UI.WebControls.WebParts.WebPartManager.WebPartManagerControlCollection.AddWebPart(WebPart webPart) at System.Web.UI.WebControls.WebParts.WebPartManager.AddWebPart(WebPart webPart) at System.Web.UI.WebControls.WebParts.WebPartManagerInternals.AddWebPart(WebPart webPart) at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartWithRetry(WebPart webPart) at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddDynamicWebPart(WebPart webPart) at Microsoft.SharePoint.WebPartPages.SPWebPartManager.LoadWebPart(WebPart aspWebPart, String zoneId, Int32 zoneIndex, Boolean isClosed) at Microsoft.SharePoint.WebPartPages.... ...SPWebPartManager.AddWebPartInternalShared(WebPart webPart) at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset, Boolean throwIfLocked) --- End of inner exception stack trace --- at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset, Boolean throwIfLocked) at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset) at Microsoft.SharePoint.WebPartPages.WebPartQuickAdd.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) My log configuration is this: <loggingConfiguration name="Logging Application Block" tracingEnabled="true" defaultCategory="General" logWarningsWhenNoCategoriesMatch="true"> <listeners> <add fileName="XAE.log" rollSizeKB="0" timeStampPattern="yyyy-MM-dd" rollFileExistsBehavior="Overwrite" rollInterval="Day" formatter="Text Formatter" header="" footer="" listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.RollingFlatFileTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=3.1.0.0, Culture=neutral" traceOutputOptions="None" type="Microsoft.Practices.EnterpriseLibrary.Logging.TraceListeners.RollingFlatFileTraceListener, Microsoft.Practices.EnterpriseLibrary.Logging, Version=3.1.0.0, Culture=neutral" name="Rolling Flat File Trace Listener" /> <add fileName="IDOL.log" rollSizeKB="0" timeStampPattern="yyyy-MM-dd" rollFileExistsBehavior="Overwrite" rollInterval="Day" formatter="Text Formatter" header="" footer="" listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.RollingFlatFileTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=3.1.0.0, Culture=neutral" traceOutputOptions="None" type="Microsoft.Practices.EnterpriseLibrary.Logging.TraceListeners.RollingFlatFileTraceListener, Microsoft.Practices.EnterpriseLibrary.Logging, Version=3.1.0.0, Culture=neutral" name="IDOL Rolling Flat File Trace Listener" /> </listeners> <formatters> <add template="{timestamp(local)} : {category} : {message}" type="Microsoft.Practices.EnterpriseLibrary.Logging.Formatters.TextFormatter, Microsoft.Practices.EnterpriseLibrary.Logging, Version=3.1.0.0, Culture=neutral" name="Text Formatter" /> </formatters> <categorySources>  <add switchValue="All" name="Communication"> <listeners> <add name="Rolling Flat File Trace Listener" /> </listeners> </add> <add switchValue="All" name="Debug"> <listeners> <add name="Rolling Flat File Trace Listener" /> </listeners> </add> <add switchValue="All" name="Exception"> <listeners> <add name="Rolling Flat File Trace Listener" /> </listeners> </add> <add switchValue="All" name="General"> <listeners> <add name="Rolling Flat File Trace Listener" /> </listeners> </add> <add switchValue="All" name="Warning"> <listeners> <add name="Rolling Flat File Trace Listener" /> </listeners> </add> <add switchValue="All" name="IDOL"> <listeners> <add name="IDOL Rolling Flat File Trace Listener" /> </listeners> </add> </categorySources> <specialSources> <allEvents switchValue="All" name="All Events" /> <notProcessed switchValue="All" name="Unprocessed Category" /> <errors switchValue="All" name="Logging Errors & Warnings"> <listeners> <add name="Rolling Flat File Trace Listener" /> </listeners> </errors> </specialSources> </loggingConfiguration> Clearly this is because it's trying to create the log files and WSS_Minimal doesn't allow this. Is there a simple way to disable all logging for now? Removing the logging is problematic as it's used in the underlying libraries. I have tried setting all switchValue="All" to "Off" but it still throws the exception even though nothing should be logged

Read the article

Android XML Parser isnt working

- by Bram

I am writing an android application with a XML parser. I have a parser that used to work but when I run it it isnt doing anything. This is my class: import java.net.URL; import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; import org.w3c.dom.Document; import org.w3c.dom.Element; import org.w3c.dom.Node; import org.w3c.dom.NodeList; import org.xml.sax.InputSource; import android.app.Activity; import android.os.Bundle; import android.widget.LinearLayout; import android.widget.TextView; import android.widget.Toast; public class XMLParsingUsingDomeActivity extends Activity { @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); LinearLayout layout = new LinearLayout(this); layout.setOrientation(1); TextView ID[]; TextView vraag[]; TextView category[]; TextView a1[]; TextView p1[]; TextView a2[]; TextView p2[]; TextView a3[]; TextView p3[]; try { URL url = new URL( "http://128.140.217.126/vragen.xml"); DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); DocumentBuilder dbu= dbf.newDocumentBuilder(); Document doc = dbu.parse(new InputSource(url.openStream())); doc.getDocumentElement().normalize(); NodeList nodeList = doc.getElementsByTagName("item"); ID = new TextView[nodeList.getLength()]; vraag = new TextView[nodeList.getLength()]; category = new TextView[nodeList.getLength()]; a1 = new TextView[nodeList.getLength()]; p1 = new TextView[nodeList.getLength()]; a2 = new TextView[nodeList.getLength()]; p2 = new TextView[nodeList.getLength()]; a3 = new TextView[nodeList.getLength()]; p3 = new TextView[nodeList.getLength()]; for (int i = 0; i < nodeList.getLength(); i++) { Node node = nodeList.item(i); ID[i] = new TextView(this); vraag[i] = new TextView(this); category[i] = new TextView(this); a1[i] = new TextView(this); p1[i] = new TextView(this); a2[i] = new TextView(this); p2[i] = new TextView(this); a3[i] = new TextView(this); p3[i] = new TextView(this); Element fstElmnt = (Element) node; NodeList nameList = fstElmnt.getElementsByTagName("ID"); Element nameElement = (Element) nameList.item(0); nameList = nameElement.getChildNodes(); ID[i].setText(((Node) nameList.item(0)).getNodeValue()); NodeList vraagList = fstElmnt.getElementsByTagName("vraag"); Element vraagElement = (Element) vraagList.item(0); vraagList = vraagElement.getChildNodes(); vraag[i].setText(((Node) vraagList.item(0)).getNodeValue()); NodeList a1List = fstElmnt.getElementsByTagName("a1"); Element a1Element = (Element) a1List.item(0); a1List = a1Element.getChildNodes(); a1[i].setText(((Node) a1List.item(0)).getNodeValue()); NodeList p1List = fstElmnt.getElementsByTagName("p1"); Element p1Element = (Element) p1List.item(0); p1List = p1Element.getChildNodes(); p1[i].setText(((Node) p1List.item(0)).getNodeValue()); NodeList a2List = fstElmnt.getElementsByTagName("a2"); Element a2Element = (Element) a2List.item(0); a2List = a2Element.getChildNodes(); a2[i].setText(((Node) a2List.item(0)).getNodeValue()); NodeList p2List = fstElmnt.getElementsByTagName("p2"); Element p2Element = (Element) p2List.item(0); p2List = p2Element.getChildNodes(); p2[i].setText(((Node) p2List.item(0)).getNodeValue()); NodeList a3List = fstElmnt.getElementsByTagName("a3"); Element a3Element = (Element) a3List.item(0); a3List = a3Element.getChildNodes(); a3[i].setText(((Node) a3List.item(0)).getNodeValue()); NodeList p3List = fstElmnt.getElementsByTagName("p3"); Element p3Element = (Element) p3List.item(0); p3List = p3Element.getChildNodes(); p3[i].setText(((Node) p3List.item(0)).getNodeValue()); layout.addView(category[i]); Toast.makeText(this, "ID: " + i + "\n" + "Vraag: " + ((Node) vraagList.item(0)).getNodeValue() + "\n" + "A1: " + ((Node) a1List.item(0)).getNodeValue() + "\n" + "P2: " + ((Node) p1List.item(0)).getNodeValue() + "\n" + "A2: " + ((Node) a2List.item(0)).getNodeValue() + "\n" + "P2: " + ((Node) p2List.item(0)).getNodeValue() + "\n" + "A3: " + ((Node) a3List.item(0)).getNodeValue() + "\n" + "P3: " + ((Node) p3List.item(0)).getNodeValue(), Toast.LENGTH_LONG).show(); } } catch (Exception e) { System.out.println("XML Pasing Excpetion = " + e); } /** Set the layout view to display */ setContentView(layout); } } And my manifest: <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="your.pace.namace" android:versionCode="1" android:versionName="1.0" > <uses-sdk android:minSdkVersion="10" /> <uses-permission android:name="android.permission.INTERNET"></uses-permission> <application android:icon="@drawable/ic_launcher" android:label="@string/app_name" > <activity android:name=".XMLParsingUsingDomeActivity" android:label="@string/app_name" > <intent-filter> <action android:name="android.intent.action.MAIN" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter> </activity> </application> </manifest> And the logcat output is worthless. I didnt change the code but its just not working anymore.

Read the article

Non recursive way to position a genogram in 2D points for x axis. Descendant are below

- by Nassign

I currently was tasked to make a genogram for a family consisting of siblings, parents with aunts and uncles with grandparents and greatgrandparents for only blood relatives. My current algorithm is using recursion. but I am wondering how to do it in non recursive way to make it more efficient. it is programmed in c# using graphics to draw on a bitmap. Current algorithm for calculating x position, the y position is by getting the generation number. public void StartCalculatePosition() { // Search the start node (The only node with targetFlg set to true) Person start = null; foreach (Person p in PersonDic.Values) { if (start == null) start = p; if (p.Targetflg) { start = p; break; } } CalcPositionRecurse(start); // Normalize the position (shift all values to positive value) // Get the minimum value (must be negative) // Then offset the position of all marriage and person with that to make it start from zero float minPosition = float.MaxValue; foreach (Person p in PersonDic.Values) { if (minPosition > p.Position) { minPosition = p.Position; } } if (minPosition < 0) { foreach (Person p in PersonDic.Values) { p.Position -= minPosition; } foreach (Marriage m in MarriageList) { m.ParentsPosition -= minPosition; m.ChildrenPosition -= minPosition; } } } /// <summary> /// Calculate position of genogram using recursion /// </summary> /// <param name="psn"></param> private void CalcPositionRecurse(Person psn) { // End the recursion if (psn.BirthMarriage == null || psn.BirthMarriage.Parents.Count == 0) { psn.Position = 0.0f; if (psn.BirthMarriage != null) { psn.BirthMarriage.ParentsPosition = 0.0f; psn.BirthMarriage.ChildrenPosition = 0.0f; } CalculateSiblingPosition(psn); return; } // Left recurse if (psn.Father != null) { CalcPositionRecurse(psn.Father); } // Right recurse if (psn.Mother != null) { CalcPositionRecurse(psn.Mother); } // Merge Position if (psn.Father != null && psn.Mother != null) { AdjustConflict(psn.Father, psn.Mother); // Position person in center of parent psn.Position = (psn.Father.Position + psn.Mother.Position) / 2; psn.BirthMarriage.ParentsPosition = psn.Position; psn.BirthMarriage.ChildrenPosition = psn.Position; } else { // Single mom or single dad if (psn.Father != null) { psn.Position = psn.Father.Position; psn.BirthMarriage.ParentsPosition = psn.Position; psn.BirthMarriage.ChildrenPosition = psn.Position; } else if (psn.Mother != null) { psn.Position = psn.Mother.Position; psn.BirthMarriage.ParentsPosition = psn.Position; psn.BirthMarriage.ChildrenPosition = psn.Position; } else { // Should not happen, checking in start of function } } // Arrange the siblings base on my position (left younger, right older) CalculateSiblingPosition(psn); } private float GetRightBoundaryAncestor(Person psn) { float rPos = psn.Position; // Get the rightmost position among siblings foreach (Person sibling in psn.Siblings) { if (sibling.Position > rPos) { rPos = sibling.Position; } } if (psn.Father != null) { float rFatherPos = GetRightBoundaryAncestor(psn.Father); if (rFatherPos > rPos) { rPos = rFatherPos; } } if (psn.Mother != null) { float rMotherPos = GetRightBoundaryAncestor(psn.Mother); if (rMotherPos > rPos) { rPos = rMotherPos; } } return rPos; } private float GetLeftBoundaryAncestor(Person psn) { float rPos = psn.Position; // Get the rightmost position among siblings foreach (Person sibling in psn.Siblings) { if (sibling.Position < rPos) { rPos = sibling.Position; } } if (psn.Father != null) { float rFatherPos = GetLeftBoundaryAncestor(psn.Father); if (rFatherPos < rPos) { rPos = rFatherPos; } } if (psn.Mother != null) { float rMotherPos = GetLeftBoundaryAncestor(psn.Mother); if (rMotherPos < rPos) { rPos = rMotherPos; } } return rPos; } /// <summary> /// Check if two parent group has conflict and compensate on the conflict /// </summary> /// <param name="leftGroup"></param> /// <param name="rightGroup"></param> public void AdjustConflict(Person leftGroup, Person rightGroup) { float leftMax = GetRightBoundaryAncestor(leftGroup); leftMax += 0.5f; float rightMin = GetLeftBoundaryAncestor(rightGroup); rightMin -= 0.5f; float diff = leftMax - rightMin; if (diff > 0.0f) { float moveHalf = Math.Abs(diff) / 2; RecurseMoveAncestor(leftGroup, 0 - moveHalf); RecurseMoveAncestor(rightGroup, moveHalf); } } /// <summary> /// Recursively move a person and all his/her ancestor /// </summary> /// <param name="psn"></param> /// <param name="moveUnit"></param> public void RecurseMoveAncestor(Person psn, float moveUnit) { psn.Position += moveUnit; foreach (Person siblings in psn.Siblings) { if (siblings.Id != psn.Id) { siblings.Position += moveUnit; } } if (psn.BirthMarriage != null) { psn.BirthMarriage.ChildrenPosition += moveUnit; psn.BirthMarriage.ParentsPosition += moveUnit; } if (psn.Father != null) { RecurseMoveAncestor(psn.Father, moveUnit); } if (psn.Mother != null) { RecurseMoveAncestor(psn.Mother, moveUnit); } } /// <summary> /// Calculate the position of the siblings /// </summary> /// <param name="psn"></param> /// <param name="anchor"></param> public void CalculateSiblingPosition(Person psn) { if (psn.Siblings.Count == 0) { return; } List<Person> sibling = psn.Siblings; int argidx; for (argidx = 0; argidx < sibling.Count; argidx++) { if (sibling[argidx].Id == psn.Id) { break; } } // Compute position for each brother that is younger that person int idx; for (idx = argidx - 1; idx >= 0; idx--) { sibling[idx].Position = sibling[idx + 1].Position - 1; } for (idx = argidx + 1; idx < sibling.Count; idx++) { sibling[idx].Position = sibling[idx - 1].Position + 1; } }

Read the article

Extending Oracle CEP with Predictive Analytics

- by vikram.shukla(at)oracle.com

Introduction: OCEP is often used as a business rules engine to execute a set of business logic rules via CQL statements, and take decisions based on the outcome of those rules. There are times where configuring rules manually is sufficient because an application needs to deal with only a small and well-defined set of static rules. However, in many situations customers don't want to pre-define such rules for two reasons. First, they are dealing with events with lots of columns and manually crafting such rules for each column or a set of columns and combinations thereof is almost impossible. Second, they are content with probabilistic outcomes and do not care about 100% precision. The former is the case when a user is dealing with data with high dimensionality, the latter when an application can live with "false" positives as they can be discarded after further inspection, say by a Human Task component in a Business Process Management software. The primary goal of this blog post is to show how this can be achieved by combining OCEP with Oracle Data Mining® and leveraging the latter's rich set of algorithms and functionality to do predictive analytics in real time on streaming events. The secondary goal of this post is also to show how OCEP can be extended to invoke any arbitrary external computation in an RDBMS from within CEP. The extensible facility is known as the JDBC cartridge. The rest of the post describes the steps required to achieve this: We use the dataset available at http://blogs.oracle.com/datamining/2010/01/fraud_and_anomaly_detection_made_simple.html to showcase the capabilities. We use it to show how transaction anomalies or fraud can be detected. Building the model: Follow the self-explanatory steps described at the above URL to build the model. It is very simple - it uses built-in Oracle Data Mining PL/SQL packages to cleanse, normalize and build the model out of the dataset. You can also use graphical Oracle Data Miner® to build the models. To summarize, it involves: Specifying which algorithms to use. In this case we use Support Vector Machines as we're trying to find anomalies in highly dimensional dataset.Build model on the data in the table for the algorithms specified. For this example, the table was populated in the scott/tiger schema with appropriate privileges. Configuring the Data Source: This is the first step in building CEP application using such an integration. Our datasource looks as follows in the server config file. It is advisable that you use the Visualizer to add it to the running server dynamically, rather than manually edit the file. <data-source> <name>DataMining</name> <data-source-params> <jndi-names> <element>DataMining</element> </jndi-names> <global-transactions-protocol>OnePhaseCommit</global-transactions-protocol> </data-source-params> <connection-pool-params> <credential-mapping-enabled></credential-mapping-enabled> <test-table-name>SQL SELECT 1 from DUAL</test-table-name> <initial-capacity>1</initial-capacity> <max-capacity>15</max-capacity> <capacity-increment>1</capacity-increment> </connection-pool-params> <driver-params> <use-xa-data-source-interface>true</use-xa-data-source-interface> <driver-name>oracle.jdbc.OracleDriver</driver-name> <url>jdbc:oracle:thin:@localhost:1522:orcl</url> <properties> <element> <value>scott</value> <name>user</name> </element> <element> <value>{Salted-3DES}AzFE5dDbO2g=</value> <name>password</name> </element> <element> <name>com.bea.core.datasource.serviceName</name> <value>oracle11.2g</value> </element> <element> <name>com.bea.core.datasource.serviceVersion</name> <value>11.2.0</value> </element> <element> <name>com.bea.core.datasource.serviceObjectClass</name> <value>java.sql.Driver</value> </element> </properties> </driver-params> </data-source> Designing the EPN: The EPN is very simple in this example. We briefly describe each of the components. The adapter ("DataMiningAdapter") reads data from a .csv file and sends it to the CQL processor downstream. The event payload here is same as that of the table in the database (refer to the attached project or do a "desc table-name" from a SQL*PLUS prompt). While this is for convenience in this example, it need not be the case. One can still omit fields in the streaming events, and need not match all columns in the table on which the model was built. Better yet, it does not even need to have the same name as columns in the table, as long as you alias them in the USING clause of the mining function. (Caveat: they still need to draw values from a similar universe or domain, otherwise it constitutes incorrect usage of the model). There are two things in the CQL processor ("DataMiningProc") that make scoring possible on streaming events. 1. User defined cartridge function Please refer to the OCEP CQL reference manual to find more details about how to define such functions. We include the function below in its entirety for illustration. <?xml version="1.0" encoding="UTF-8"?> <jdbcctxconfig:config xmlns:jdbcctxconfig="http://www.bea.com/ns/wlevs/config/application" xmlns:jc="http://www.oracle.com/ns/ocep/config/jdbc"> <jc:jdbc-ctx> <name>Oracle11gR2</name> <data-source>DataMining</data-source> <function name="prediction2"> <param name="CQLMONTH" type="char"/> <param name="WEEKOFMONTH" type="int"/> <param name="DAYOFWEEK" type="char" /> <param name="MAKE" type="char" /> <param name="ACCIDENTAREA" type="char" /> <param name="DAYOFWEEKCLAIMED" type="char" /> <param name="MONTHCLAIMED" type="char" /> <param name="WEEKOFMONTHCLAIMED" type="int" /> <param name="SEX" type="char" /> <param name="MARITALSTATUS" type="char" /> <param name="AGE" type="int" /> <param name="FAULT" type="char" /> <param name="POLICYTYPE" type="char" /> <param name="VEHICLECATEGORY" type="char" /> <param name="VEHICLEPRICE" type="char" /> <param name="FRAUDFOUND" type="int" /> <param name="POLICYNUMBER" type="int" /> <param name="REPNUMBER" type="int" /> <param name="DEDUCTIBLE" type="int" /> <param name="DRIVERRATING" type="int" /> <param name="DAYSPOLICYACCIDENT" type="char" /> <param name="DAYSPOLICYCLAIM" type="char" /> <param name="PASTNUMOFCLAIMS" type="char" /> <param name="AGEOFVEHICLES" type="char" /> <param name="AGEOFPOLICYHOLDER" type="char" /> <param name="POLICEREPORTFILED" type="char" /> <param name="WITNESSPRESNT" type="char" /> <param name="AGENTTYPE" type="char" /> <param name="NUMOFSUPP" type="char" /> <param name="ADDRCHGCLAIM" type="char" /> <param name="NUMOFCARS" type="char" /> <param name="CQLYEAR" type="int" /> <param name="BASEPOLICY" type="char" /> <return-component-type>char</return-component-type> <sql><![CDATA[ SELECT to_char(PREDICTION_PROBABILITY(CLAIMSMODEL, '0' USING *)) AS probability FROM (SELECT :CQLMONTH AS MONTH, :WEEKOFMONTH AS WEEKOFMONTH, :DAYOFWEEK AS DAYOFWEEK, :MAKE AS MAKE, :ACCIDENTAREA AS ACCIDENTAREA, :DAYOFWEEKCLAIMED AS DAYOFWEEKCLAIMED, :MONTHCLAIMED AS MONTHCLAIMED, :WEEKOFMONTHCLAIMED, :SEX AS SEX, :MARITALSTATUS AS MARITALSTATUS, :AGE AS AGE, :FAULT AS FAULT, :POLICYTYPE AS POLICYTYPE, :VEHICLECATEGORY AS VEHICLECATEGORY, :VEHICLEPRICE AS VEHICLEPRICE, :FRAUDFOUND AS FRAUDFOUND, :POLICYNUMBER AS POLICYNUMBER, :REPNUMBER AS REPNUMBER, :DEDUCTIBLE AS DEDUCTIBLE, :DRIVERRATING AS DRIVERRATING, :DAYSPOLICYACCIDENT AS DAYSPOLICYACCIDENT, :DAYSPOLICYCLAIM AS DAYSPOLICYCLAIM, :PASTNUMOFCLAIMS AS PASTNUMOFCLAIMS, :AGEOFVEHICLES AS AGEOFVEHICLES, :AGEOFPOLICYHOLDER AS AGEOFPOLICYHOLDER, :POLICEREPORTFILED AS POLICEREPORTFILED, :WITNESSPRESNT AS WITNESSPRESENT, :AGENTTYPE AS AGENTTYPE, :NUMOFSUPP AS NUMOFSUPP, :ADDRCHGCLAIM AS ADDRCHGCLAIM, :NUMOFCARS AS NUMOFCARS, :CQLYEAR AS YEAR, :BASEPOLICY AS BASEPOLICY FROM dual) ]]> </sql> </function> </jc:jdbc-ctx> </jdbcctxconfig:config> 2. Invoking the function for each event. Once this function is defined, you can invoke it from CQL as follows: <?xml version="1.0" encoding="UTF-8"?> <wlevs:config xmlns:wlevs="http://www.bea.com/ns/wlevs/config/application"> <processor> <name>DataMiningProc</name> <rules> <query id="q1"><![CDATA[ ISTREAM(SELECT S.CQLMONTH, S.WEEKOFMONTH, S.DAYOFWEEK, S.MAKE, : S.BASEPOLICY, C.F AS probability FROM StreamDataChannel [NOW] AS S, TABLE(prediction2@Oracle11gR2(S.CQLMONTH, S.WEEKOFMONTH, S.DAYOFWEEK, S.MAKE, ..., S.BASEPOLICY) AS F of char) AS C) ]]></query> </rules> </processor> </wlevs:config> Finally, the last stage in the EPN prints out the probability of the event being an anomaly. One can also define a threshold in CQL to filter out events that are normal, i.e., below a certain mark as defined by the analyst or designer. Sample Runs: Now let's see how this behaves when events are streamed through CEP. We use only two events for brevity, one normal and other one not. This is one of the "normal" looking events and the probability of it being anomalous is less than 60%. Event is: eventType=DataMiningOutEvent object=q1 time=2904821976256 S.CQLMONTH=Dec, S.WEEKOFMONTH=5, S.DAYOFWEEK=Wednesday, S.MAKE=Honda, S.ACCIDENTAREA=Urban, S.DAYOFWEEKCLAIMED=Tuesday, S.MONTHCLAIMED=Jan, S.WEEKOFMONTHCLAIMED=1, S.SEX=Female, S.MARITALSTATUS=Single, S.AGE=21, S.FAULT=Policy Holder, S.POLICYTYPE=Sport - Liability, S.VEHICLECATEGORY=Sport, S.VEHICLEPRICE=more than 69000, S.FRAUDFOUND=0, S.POLICYNUMBER=1, S.REPNUMBER=12, S.DEDUCTIBLE=300, S.DRIVERRATING=1, S.DAYSPOLICYACCIDENT=more than 30, S.DAYSPOLICYCLAIM=more than 30, S.PASTNUMOFCLAIMS=none, S.AGEOFVEHICLES=3 years, S.AGEOFPOLICYHOLDER=26 to 30, S.POLICEREPORTFILED=No, S.WITNESSPRESENT=No, S.AGENTTYPE=External, S.NUMOFSUPP=none, S.ADDRCHGCLAIM=1 year, S.NUMOFCARS=3 to 4, S.CQLYEAR=1994, S.BASEPOLICY=Liability, probability=.58931702982118561 isTotalOrderGuarantee=true\nAnamoly probability: .58931702982118561 However, the following event is scored as an anomaly with a very high probability of 89%. So there is likely to be something wrong with it. A close look reveals that the value of "deductible" field (10000) is not "normal". What exactly constitutes normal here?. If you run the query on the database to find ALL distinct values for the "deductible" field, it returns the following set: {300, 400, 500, 700} Event is: eventType=DataMiningOutEvent object=q1 time=2598483773496 S.CQLMONTH=Dec, S.WEEKOFMONTH=5, S.DAYOFWEEK=Wednesday, S.MAKE=Honda, S.ACCIDENTAREA=Urban, S.DAYOFWEEKCLAIMED=Tuesday, S.MONTHCLAIMED=Jan, S.WEEKOFMONTHCLAIMED=1, S.SEX=Female, S.MARITALSTATUS=Single, S.AGE=21, S.FAULT=Policy Holder, S.POLICYTYPE=Sport - Liability, S.VEHICLECATEGORY=Sport, S.VEHICLEPRICE=more than 69000, S.FRAUDFOUND=0, S.POLICYNUMBER=1, S.REPNUMBER=12, S.DEDUCTIBLE=10000, S.DRIVERRATING=1, S.DAYSPOLICYACCIDENT=more than 30, S.DAYSPOLICYCLAIM=more than 30, S.PASTNUMOFCLAIMS=none, S.AGEOFVEHICLES=3 years, S.AGEOFPOLICYHOLDER=26 to 30, S.POLICEREPORTFILED=No, S.WITNESSPRESENT=No, S.AGENTTYPE=External, S.NUMOFSUPP=none, S.ADDRCHGCLAIM=1 year, S.NUMOFCARS=3 to 4, S.CQLYEAR=1994, S.BASEPOLICY=Liability, probability=.89171554529576691 isTotalOrderGuarantee=true\nAnamoly probability: .89171554529576691 Conclusion: By way of this example, we show: real-time scoring of events as they flow through CEP leveraging Oracle Data Mining.how CEP applications can invoke complex arbitrary external computations (function shipping) in an RDBMS.

Read the article

compass-rails 1.03 - TypeError: can't convert nil into String

- by Romiko

I am running: ruby 1.9.3p392 (2013-02-22) [i386-mingw32] compass-rails 1.0.3 I used the Windows RailsInstaller to install Ruby on Rails Gemfile group :assets do gem 'sass-rails', '~> 3.2.3' gem 'coffee-rails', '~> 3.2.1' gem 'compass-rails','~> 1.0.2' # See https://github.com/sstephenson/execjs#readme for more supported runtimes # gem 'therubyracer', :platforms => :ruby gem 'uglifier', '>= 1.0.3' end I am currently experiencing issues importing sprites. My sprites are in: assets/images/source in my _shared.scss file I have: //Sprites @import "./source/*.png"; $source-sprite-dimensions: true; In my application.scss I have: /* * This is a manifest file that'll be compiled into application.css, which will include all the files * listed below. * * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets, * or vendor/assets/stylesheets of plugins, if any, can be referenced here using a relative path. * * You're free to add application-wide styles to this file and they'll appear at the top of the * compiled file, but it's generally better to create a new file per style scope. * *= require_self */ @import "_shared.scss"; @import "baseline.scss"; @import "global.scss"; @import "normalize.scss"; @import "print.scss"; @import "desktop.scss"; @import "tablet.scss"; @import "home.css.scss"; I am also using rails server and not compass watcher. However when I browse to the page at localhost:3000/assets/application.css, I get the following error: body:before { font-weight: bold; content: "\000a TypeError: can't convert nil into String\000a (in c:\002f RangerRomOnRails\002f RangerRom\002f app\002f assets\002f stylesheets\002f desktop.scss)"; } body:after { content: "\000a C:\002f RailsInstaller\002f Ruby1.9.3\002f lib\002f ruby\002f gems\002f 1.9.1\002f gems\002f compass-0.12.2\002f lib\002f compass\002f sass_extensions\002f functions\002f image_size.rb:17:in `extname'"; } Here is the full stack trace: compass (0 .12.2) lib/compass/sass_extensions/functions/image_size.rb:17:in `extname' compass (0.12.2) lib/compass/sass_extensions/functions/image_size.rb:17:in `initialize' compass (0.12.2) lib/compass/sass_extensions/functions/image_size.rb:50:in `new' compass (0.12.2) lib/compass/sass_extensions/functions/image_size.rb:50:in `image_dimensions' compass (0.12.2) lib/compass/sass_extensions/functions/image_size.rb:4:in `image_width' sass (3.2.9) lib/sass/script/funcall.rb:112:in `_perform' sass (3.2.9) lib/sass/script/node.rb:40:in `perform' sass (3.2.9) lib/sass/tree/visitors/perform.rb:298:in `visit_prop' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:100:in `visit' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `map' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:109:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:121:in `with_environment' sass (3.2.9) lib/sass/tree/visitors/perform.rb:108:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `block in visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:320:in `visit_rule' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:100:in `visit' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `map' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:109:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:121:in `with_environment' sass (3.2.9) lib/sass/tree/visitors/perform.rb:108:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `block in visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:320:in `visit_rule' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:100:in `visit' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `map' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:109:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:121:in `with_environment' sass (3.2.9) lib/sass/tree/visitors/perform.rb:108:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `block in visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:362:in `visit_media' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:100:in `visit' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `map' sass (3.2.9) lib/sass/tree/visitors/base.rb:53:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:109:in `block in visit_children' sass (3.2.9) lib/sass/tree/visitors/perform.rb:121:in `with_environment' sass (3.2.9) lib/sass/tree/visitors/perform.rb:108:in `visit_children' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `block in visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:128:in `visit_root' sass (3.2.9) lib/sass/tree/visitors/base.rb:37:in `visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:100:in `visit' sass (3.2.9) lib/sass/tree/visitors/perform.rb:7:in `visit' sass (3.2.9) lib/sass/tree/root_node.rb:20:in `render' sass (3.2.9) lib/sass/engine.rb:315:in `_render' sass (3.2.9) lib/sass/engine.rb:262:in `render' sass-rails (3.2.6) lib/sass/rails/template_handlers.rb:106:in `evaluate' tilt (1.4.1) lib/tilt/template.rb:103:in `render' sprockets (2.2.2) lib/sprockets/context.rb:193:in `block in evaluate' sprockets (2.2.2) lib/sprockets/context.rb:190:in `each' sprockets (2.2.2) lib/sprockets/context.rb:190:in `evaluate' sprockets (2.2.2) lib/sprockets/processed_asset.rb:12:in `initialize' sprockets (2.2.2) lib/sprockets/base.rb:249:in `new' sprockets (2.2.2) lib/sprockets/base.rb:249:in `block in build_asset' sprockets (2.2.2) lib/sprockets/base.rb:270:in `circular_call_protection' sprockets (2.2.2) lib/sprockets/base.rb:248:in `build_asset' sprockets (2.2.2) lib/sprockets/index.rb:93:in `block in build_asset' sprockets (2.2.2) lib/sprockets/caching.rb:19:in `cache_asset' sprockets (2.2.2) lib/sprockets/index.rb:92:in `build_asset' sprockets (2.2.2) lib/sprockets/base.rb:169:in `find_asset' sprockets (2.2.2) lib/sprockets/index.rb:60:in `find_asset' sprockets (2.2.2) lib/sprockets/processed_asset.rb:111:in `block in resolve_dependencies' sprockets (2.2.2) lib/sprockets/processed_asset.rb:105:in `each' sprockets (2.2.2) lib/sprockets/processed_asset.rb:105:in `resolve_dependencies' sprockets (2.2.2) lib/sprockets/processed_asset.rb:97:in `build_required_assets' sprockets (2.2.2) lib/sprockets/processed_asset.rb:16:in `initialize' sprockets (2.2.2) lib/sprockets/base.rb:249:in `new' sprockets (2.2.2) lib/sprockets/base.rb:249:in `block in build_asset' sprockets (2.2.2) lib/sprockets/base.rb:270:in `circular_call_protection' sprockets (2.2.2) lib/sprockets/base.rb:248:in `build_asset' sprockets (2.2.2) lib/sprockets/index.rb:93:in `block in build_asset' sprockets (2.2.2) lib/sprockets/caching.rb:19:in `cache_asset' sprockets (2.2.2) lib/sprockets/index.rb:92:in `build_asset' sprockets (2.2.2) lib/sprockets/base.rb:169:in `find_asset' sprockets (2.2.2) lib/sprockets/index.rb:60:in `find_asset' sprockets (2.2.2) lib/sprockets/bundled_asset.rb:38:in `init_with' sprockets (2.2.2) lib/sprockets/asset.rb:24:in `from_hash' sprockets (2.2.2) lib/sprockets/caching.rb:15:in `cache_asset' sprockets (2.2.2) lib/sprockets/index.rb:92:in `build_asset' sprockets (2.2.2) lib/sprockets/base.rb:169:in `find_asset' sprockets (2.2.2) lib/sprockets/index.rb:60:in `find_asset' sprockets (2.2.2) lib/sprockets/environment.rb:78:in `find_asset' sprockets (2.2.2) lib/sprockets/base.rb:177:in `[]' actionpack (3.2.13) lib/sprockets/helpers/rails_helper.rb:126:in `asset_for' actionpack (3.2.13) lib/sprockets/helpers/rails_helper.rb:44:in `block in stylesheet_link_tag' actionpack (3.2.13) lib/sprockets/helpers/rails_helper.rb:43:in `collect' actionpack (3.2.13) lib/sprockets/helpers/rails_helper.rb:43:in `stylesheet_link_tag' app/views/layouts/application.html.erb:16:in `_app_views_layouts_application_html_erb___824639613_33845076' actionpack (3.2.13) lib/action_view/template.rb:145:in `block in render' activesupport (3.2.13) lib/active_support/notifications.rb:125:in `instrument' actionpack (3.2.13) lib/action_view/template.rb:143:in `render' actionpack (3.2.13) lib/action_view/renderer/template_renderer.rb:59:in `render_with_layout' actionpack (3.2.13) lib/action_view/renderer/template_renderer.rb:45:in `render_template' actionpack (3.2.13) lib/action_view/renderer/template_renderer.rb:18:in `render' actionpack (3.2.13) lib/action_view/renderer/renderer.rb:36:in `render_template' actionpack (3.2.13) lib/action_view/renderer/renderer.rb:17:in `render' actionpack (3.2.13) lib/abstract_controller/rendering.rb:110:in `_render_template' actionpack (3.2.13) lib/action_controller/metal/streaming.rb:225:in `_render_template' actionpack (3.2.13) lib/abstract_controller/rendering.rb:103:in `render_to_body' actionpack (3.2.13) lib/action_controller/metal/renderers.rb:28:in `render_to_body' actionpack (3.2.13) lib/action_controller/metal/compatibility.rb:50:in `render_to_body' actionpack (3.2.13) lib/abstract_controller/rendering.rb:88:in `render' actionpack (3.2.13) lib/action_controller/metal/rendering.rb:16:in `render' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:40:in `block (2 levels) in render' activesupport (3.2.13) lib/active_support/core_ext/benchmark.rb:5:in `block in ms' C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' activesupport (3.2.13) lib/active_support/core_ext/benchmark.rb:5:in `ms' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:40:in `block in render' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:83:in `cleanup_view_runtime' activerecord (3.2.13) lib/active_record/railties/controller_runtime.rb:24:in `cleanup_view_runtime' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:39:in `render' actionpack (3.2.13) lib/action_controller/metal/implicit_render.rb:10:in `default_render' actionpack (3.2.13) lib/action_controller/metal/implicit_render.rb:5:in `send_action' actionpack (3.2.13) lib/abstract_controller/base.rb:167:in `process_action' actionpack (3.2.13) lib/action_controller/metal/rendering.rb:10:in `process_action' actionpack (3.2.13) lib/abstract_controller/callbacks.rb:18:in `block in process_action' activesupport (3.2.13) lib/active_support/callbacks.rb:414:in `_run__956028316__process_action__416811168__callbacks' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `__run_callback' activesupport (3.2.13) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' activesupport (3.2.13) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (3.2.13) lib/abstract_controller/callbacks.rb:17:in `process_action' actionpack (3.2.13) lib/action_controller/metal/rescue.rb:29:in `process_action' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' activesupport (3.2.13) lib/active_support/notifications.rb:123:in `block in instrument' activesupport (3.2.13) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (3.2.13) lib/active_support/notifications.rb:123:in `instrument' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:29:in `process_action' actionpack (3.2.13) lib/action_controller/metal/params_wrapper.rb:207:in `process_action' activerecord (3.2.13) lib/active_record/railties/controller_runtime.rb:18:in `process_action' actionpack (3.2.13) lib/abstract_controller/base.rb:121:in `process' actionpack (3.2.13) lib/abstract_controller/rendering.rb:45:in `process' actionpack (3.2.13) lib/action_controller/metal.rb:203:in `dispatch' actionpack (3.2.13) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' actionpack (3.2.13) lib/action_controller/metal.rb:246:in `block in action' actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:73:in `call' actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:73:in `dispatch' actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:36:in `call' journey (1.0.4) lib/journey/router.rb:68:in `block in call' journey (1.0.4) lib/journey/router.rb:56:in `each' journey (1.0.4) lib/journey/router.rb:56:in `call' actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:612:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' rack (1.4.5) lib/rack/etag.rb:23:in `call' rack (1.4.5) lib/rack/conditionalget.rb:25:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/head.rb:14:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/params_parser.rb:21:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/flash.rb:242:in `call' rack (1.4.5) lib/rack/session/abstract/id.rb:210:in `context' rack (1.4.5) lib/rack/session/abstract/id.rb:205:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/cookies.rb:341:in `call' activerecord (3.2.13) lib/active_record/query_cache.rb:64:in `call' activerecord (3.2.13) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `_run__360878605__call__248365880__callbacks' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `__run_callback' activesupport (3.2.13) lib/active_support/callbacks.rb:385:in `_run_call_callbacks' activesupport (3.2.13) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:27:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/reloader.rb:65:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/remote_ip.rb:31:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' railties (3.2.13) lib/rails/rack/logger.rb:32:in `call_app' railties (3.2.13) lib/rails/rack/logger.rb:16:in `block in call' activesupport (3.2.13) lib/active_support/tagged_logging.rb:22:in `tagged' railties (3.2.13) lib/rails/rack/logger.rb:16:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/request_id.rb:22:in `call' rack (1.4.5) lib/rack/methodoverride.rb:21:in `call' rack (1.4.5) lib/rack/runtime.rb:17:in `call' activesupport (3.2.13) lib/active_support/cache/strategy/local_cache.rb:72:in `call' rack (1.4.5) lib/rack/lock.rb:15:in `call' actionpack (3.2.13) lib/action_dispatch/middleware/static.rb:63:in `call' railties (3.2.13) lib/rails/engine.rb:479:in `call' railties (3.2.13) lib/rails/application.rb:223:in `call' rack (1.4.5) lib/rack/content_length.rb:14:in `call' railties (3.2.13) lib/rails/rack/log_tailer.rb:17:in `call' rack (1.4.5) lib/rack/handler/webrick.rb:59:in `service' C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/webrick/httpserver.rb:138:in `service' C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/webrick/httpserver.rb:94:in `run' C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'

Read the article

Reading Data from DDFS ValueError: No JSON object could be decoded

- by secumind

I'm running dozens of map reduce jobs for a number of different purposes using disco. My data has grown enormous and I thought I would try using DDFS for a change rather than standard txt files. I've followed the DISCO map/reduce example Counting Words as a map/reduce job, without to much difficulty and with the help of others, Reading JSON specific data into DISCO I've gotten past one of my latest problems. I'm trying to read data in/out of ddfs to better chunk and distribute it but am having a bit of trouble. Here's an example file: file.txt {"favorited": false, "in_reply_to_user_id": null, "contributors": null, "truncated": false, "text": "I'll call him back tomorrow I guess", "created_at": "Mon Feb 13 05:34:27 +0000 2012", "retweeted": false, "in_reply_to_status_id_str": null, "coordinates": null, "in_reply_to_user_id_str": null, "entities": {"user_mentions": [], "hashtags": [], "urls": []}, "in_reply_to_status_id": null, "id_str": "168931016843603968", "place": null, "user": {"follow_request_sent": null, "profile_use_background_image": true, "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/305726905/FASHION-3.png", "verified": false, "profile_image_url_https": "https://si0.twimg.com/profile_images/1818996723/image_normal.jpg", "profile_sidebar_fill_color": "292727", "is_translator": false, "id": 113532729, "profile_text_color": "000000", "followers_count": 78, "protected": false, "location": "With My Niggas In Paris!", "default_profile_image": false, "listed_count": 0, "utc_offset": -21600, "statuses_count": 6733, "description": "Made in CHINA., Educated && Making My Own $$. Fear GOD && Put Him 1st. #TeamFollowBack #TeamiPhone\n", "friends_count": 74, "profile_link_color": "b03f3f", "profile_image_url": "http://a2.twimg.com/profile_images/1818996723/image_normal.jpg", "notifications": null, "show_all_inline_media": false, "geo_enabled": true, "profile_background_color": "1f9199", "id_str": "113532729", "profile_background_image_url": "http://a3.twimg.com/profile_background_images/305726905/FASHION-3.png", "name": "Bee'Jay", "lang": "en", "profile_background_tile": true, "favourites_count": 19, "screen_name": "OohMyBEEsNice", "url": "http://www.bitchimpaid.org", "created_at": "Fri Feb 12 03:32:54 +0000 2010", "contributors_enabled": false, "time_zone": "Central Time (US & Canada)", "profile_sidebar_border_color": "000000", "default_profile": false, "following": null}, "in_reply_to_screen_name": null, "retweet_count": 0, "geo": null, "id": 168931016843603968, "source": "<a href=\"http://twitter.com/#!/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>"} {"favorited": false, "in_reply_to_user_id": 50940453, "contributors": null, "truncated": false, "text": "@LegaMrvica @MimozaBand makasi om artis :D kadoo kadoo", "created_at": "Mon Feb 13 05:34:27 +0000 2012", "retweeted": false, "in_reply_to_status_id_str": "168653037894770688", "coordinates": null, "in_reply_to_user_id_str": "50940453", "entities": {"user_mentions": [{"indices": [0, 11], "screen_name": "LegaMrvica", "id": 50940453, "name": "Lega_thePianis", "id_str": "50940453"}, {"indices": [12, 23], "screen_name": "MimozaBand", "id": 375128905, "name": "Mimoza", "id_str": "375128905"}], "hashtags": [], "urls": []}, "in_reply_to_status_id": 168653037894770688, "id_str": "168931016868761600", "place": null, "user": {"follow_request_sent": null, "profile_use_background_image": true, "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/347686061/Galungan_dan_Kuningan.jpg", "verified": false, "profile_image_url_https": "https://si0.twimg.com/profile_images/1803845596/Picture_20124_normal.jpg", "profile_sidebar_fill_color": "DDFFCC", "is_translator": false, "id": 48293450, "profile_text_color": "333333", "followers_count": 182, "protected": false, "location": "\u00dcT: -6.906799,107.622383", "default_profile_image": false, "listed_count": 0, "utc_offset": -28800, "statuses_count": 3052, "description": "Fashion design maranatha '11 // traditional dancer (bali) at sanggar tampak siring & Natya Nataraja", "friends_count": 206, "profile_link_color": "0084B4", "profile_image_url": "http://a3.twimg.com/profile_images/1803845596/Picture_20124_normal.jpg", "notifications": null, "show_all_inline_media": false, "geo_enabled": true, "profile_background_color": "9AE4E8", "id_str": "48293450", "profile_background_image_url": "http://a0.twimg.com/profile_background_images/347686061/Galungan_dan_Kuningan.jpg", "name": "nana afiff", "lang": "en", "profile_background_tile": true, "favourites_count": 2, "screen_name": "hasnfebria", "url": null, "created_at": "Thu Jun 18 08:50:29 +0000 2009", "contributors_enabled": false, "time_zone": "Pacific Time (US & Canada)", "profile_sidebar_border_color": "BDDCAD", "default_profile": false, "following": null}, "in_reply_to_screen_name": "LegaMrvica", "retweet_count": 0, "geo": null, "id": 168931016868761600, "source": "<a href=\"http://blackberry.com/twitter\" rel=\"nofollow\">Twitter for BlackBerry\u00ae</a>"} {"favorited": false, "in_reply_to_user_id": 27260086, "contributors": null, "truncated": false, "text": "@justinbieber u were born to be somebody, and u're super important in beliebers' life. thanks for all biebs. I love u. follow me? 84", "created_at": "Mon Feb 13 05:34:27 +0000 2012", "retweeted": false, "in_reply_to_status_id_str": null, "coordinates": null, "in_reply_to_user_id_str": "27260086", "entities": {"user_mentions": [{"indices": [0, 13], "screen_name": "justinbieber", "id": 27260086, "name": "Justin Bieber", "id_str": "27260086"}], "hashtags": [], "urls": []}, "in_reply_to_status_id": null, "id_str": "168931016856178688", "place": null, "user": {"follow_request_sent": null, "profile_use_background_image": true, "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/416005864/Captura.JPG", "verified": false, "profile_image_url_https": "https://si0.twimg.com/profile_images/1808883280/Captura6_normal.JPG", "profile_sidebar_fill_color": "f5e7f3", "is_translator": false, "id": 406750700, "profile_text_color": "333333", "followers_count": 1122, "protected": false, "location": "Adentro de una supra.", "default_profile_image": false, "listed_count": 0, "utc_offset": -14400, "statuses_count": 20966, "description": "Mi \u00eddolo es @justinbieber , si te gusta \u00a1genial!, si no, solo respetalo. El cambi\u00f3 mi vida completamente y mi sue\u00f1o es conocerlo #TrueBelieber . ", "friends_count": 1015, "profile_link_color": "9404b8", "profile_image_url": "http://a1.twimg.com/profile_images/1808883280/Captura6_normal.JPG", "notifications": null, "show_all_inline_media": false, "geo_enabled": false, "profile_background_color": "f9fcfa", "id_str": "406750700", "profile_background_image_url": "http://a3.twimg.com/profile_background_images/416005864/Captura.JPG", "name": "neversaynever,right?", "lang": "es", "profile_background_tile": false, "favourites_count": 22, "screen_name": "True_Belieebers", "url": "http://www.wehavebieber-fever.tumblr.com", "created_at": "Mon Nov 07 04:17:40 +0000 2011", "contributors_enabled": false, "time_zone": "Santiago", "profile_sidebar_border_color": "C0DEED", "default_profile": false, "following": null}, "in_reply_to_screen_name": "justinbieber", "retweet_count": 0, "geo": null, "id": 168931016856178688, "source": "<a href=\"http://yfrog.com\" rel=\"nofollow\">Yfrog</a>"} I load it into DDFS with: # ddfs chunk data:test1 ./file.txt created: disco://localhost/ddfs/vol0/blob/44/file_txt-0$549-db27b-125e1 I test that the file is indeed loaded into ddfs with: # ddfs xcat data:test1 {"favorited": false, "in_reply_to_user_id": null, "contributors": null, "truncated": false, "text": "I'll call him back tomorrow I guess", "created_at": "Mon Feb 13 05:34:27 +0000 2012", "retweeted": false, "in_reply_to_status_id_str": null, "coordinates": null, "in_reply_to_user_id_str": null, "entities": {"user_mentions": [], "hashtags": [], "urls": []}, "in_reply_to_status_id": null, "id_str": "168931016843603968", "place": null, "user": {"follow_request_sent": null, "profile_use_background_image": true, "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/305726905/FASHION-3.png", "verified": false, "profile_image_url_https": "https://si0.twimg.com/profile_images/1818996723/image_normal.jpg", "profile_sidebar_fill_color": "292727", "is_translator": false, "id": 113532729, "profile_text_color": "000000", "followers_count": 78, "protected": false, "location": "With My Niggas In Paris!", "default_profile_image": false, "listed_count": 0, "utc_offset": -21600, "statuses_count": 6733, "description": "Made in CHINA., Educated && Making My Own $$. Fear GOD && Put Him 1st. #TeamFollowBack #TeamiPhone\n", "friends_count": 74, "profile_link_color": "b03f3f", "profile_image_url": "http://a2.twimg.com/profile_images/1818996723/image_normal.jpg", "notifications": null, "show_all_inline_media": false, "geo_enabled": true, "profile_background_color": "1f9199", "id_str": "113532729", "profile_background_image_url": "http://a3.twimg.com/profile_background_images/305726905/FASHION-3.png", "name": "Bee'Jay", "lang": "en", "profile_background_tile": true, "favourites_count": 19, "screen_name": "OohMyBEEsNice", "url": "http://www.bitchimpaid.org", "created_at": "Fri Feb 12 03:32:54 +0000 2010", "contributors_enabled": false, "time_zone": "Central Time (US & Canada)", "profile_sidebar_border_color": "000000", "default_profile": false, "following": null}, "in_reply_to_screen_name": null, "retweet_count": 0, "geo": null, "id": 168931016843603968, "source": "<a href=\"http://twitter.com/#!/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>"} {"favorited": false, "in_reply_to_user_id": 50940453, "contributors": null, "truncated": false, "text": "@LegaMrvica @MimozaBand makasi om artis :D kadoo kadoo", "created_at": "Mon Feb 13 05:34:27 +0000 2012", "retweeted": false, "in_reply_to_status_id_str": "168653037894770688", "coordinates": null, "in_reply_to_user_id_str": "50940453", "entities": {"user_mentions": [{"indices": [0, 11], "screen_name": "LegaMrvica", "id": 50940453, "name": "Lega_thePianis", "id_str": "50940453"}, {"indices": [12, 23], "screen_name": "MimozaBand", "id": 375128905, "name": "Mimoza", "id_str": "375128905"}], "hashtags": [], "urls": []}, "in_reply_to_status_id": 168653037894770688, "id_str": "168931016868761600", "place": null, "user": {"follow_request_sent": null, "profile_use_background_image": true, "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/347686061/Galungan_dan_Kuningan.jpg", "verified": false, "profile_image_url_https": "https://si0.twimg.com/profile_images/1803845596/Picture_20124_normal.jpg", "profile_sidebar_fill_color": "DDFFCC", "is_translator": false, "id": 48293450, "profile_text_color": "333333", "followers_count": 182, "protected": false, "location": "\u00dcT: -6.906799,107.622383", "default_profile_image": false, "listed_count": 0, "utc_offset": -28800, "statuses_count": 3052, "description": "Fashion design maranatha '11 // traditional dancer (bali) at sanggar tampak siring & Natya Nataraja", "friends_count": 206, "profile_link_color": "0084B4", "profile_image_url": "http://a3.twimg.com/profile_images/1803845596/Picture_20124_normal.jpg", "notifications": null, "show_all_inline_media": false, "geo_enabled": true, "profile_background_color": "9AE4E8", "id_str": "48293450", "profile_background_image_url": "http://a0.twimg.com/profile_background_images/347686061/Galungan_dan_Kuningan.jpg", "name": "nana afiff", "lang": "en", "profile_background_tile": true, "favourites_count": 2, "screen_name": "hasnfebria", "url": null, "created_at": "Thu Jun 18 08:50:29 +0000 2009", "contributors_enabled": false, "time_zone": "Pacific Time (US & Canada)", "profile_sidebar_border_color": "BDDCAD", "default_profile": false, "following": null}, "in_reply_to_screen_name": "LegaMrvica", "retweet_count": 0, "geo": null, "id": 168931016868761600, "source": "<a href=\"http://blackberry.com/twitter\" rel=\"nofollow\">Twitter for BlackBerry\u00ae</a>"} {"favorited": false, "in_reply_to_user_id": 27260086, "contributors": null, "truncated": false, "text": "@justinbieber u were born to be somebody, and u're super important in beliebers' life. thanks for all biebs. I love u. follow me? 84", "created_at": "Mon Feb 13 05:34:27 +0000 2012", "retweeted": false, "in_reply_to_status_id_str": null, "coordinates": null, "in_reply_to_user_id_str": "27260086", "entities": {"user_mentions": [{"indices": [0, 13], "screen_name": "justinbieber", "id": 27260086, "name": "Justin Bieber", "id_str": "27260086"}], "hashtags": [], "urls": []}, "in_reply_to_status_id": null, "id_str": "168931016856178688", "place": null, "user": {"follow_request_sent": null, "profile_use_background_image": true, "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/416005864/Captura.JPG", "verified": false, "profile_image_url_https": "https://si0.twimg.com/profile_images/1808883280/Captura6_normal.JPG", "profile_sidebar_fill_color": "f5e7f3", "is_translator": false, "id": 406750700, "profile_text_color": "333333", "followers_count": 1122, "protected": false, "location": "Adentro de una supra.", "default_profile_image": false, "listed_count": 0, "utc_offset": -14400, "statuses_count": 20966, "description": "Mi \u00eddolo es @justinbieber , si te gusta \u00a1genial!, si no, solo respetalo. El cambi\u00f3 mi vida completamente y mi sue\u00f1o es conocerlo #TrueBelieber . ", "friends_count": 1015, "profile_link_color": "9404b8", "profile_image_url": "http://a1.twimg.com/profile_images/1808883280/Captura6_normal.JPG", "notifications": null, "show_all_inline_media": false, "geo_enabled": false, "profile_background_color": "f9fcfa", "id_str": "406750700", "profile_background_image_url": "http://a3.twimg.com/profile_background_images/416005864/Captura.JPG", "name": "neversaynever,right?", "lang": "es", "profile_background_tile": false, "favourites_count": 22, "screen_name": "True_Belieebers", "url": "http://www.wehavebieber-fever.tumblr.com", "created_at": "Mon Nov 07 04:17:40 +0000 2011", "contributors_enabled": false, "time_zone": "Santiago", "profile_sidebar_border_color": "C0DEED", "default_profile": false, "following": null}, "in_reply_to_screen_name": "justinbieber", "retweet_count": 0, "geo": null, "id": 168931016856178688, "source": "<a href=\"http://yfrog.com\" rel=\"nofollow\">Yfrog</a> At this point everything is great, I load up the script that resulted from a previous Stack Post: from disco.core import Job, result_iterator import gzip def map(line, params): import unicodedata import json r = json.loads(line).get('text') s = unicodedata.normalize('NFD', r).encode('ascii', 'ignore') for word in s.split(): yield word, 1 def reduce(iter, params): from disco.util import kvgroup for word, counts in kvgroup(sorted(iter)): yield word, sum(counts) if __name__ == '__main__': job = Job().run(input=["tag://data:test1"], map=map, reduce=reduce) for word, count in result_iterator(job.wait(show=True)): print word, count NOTE: That this script runs file if the input=["file.txt"], however when I run it with "tag://data:test1" I get the following error: # DISCO_EVENTS=1 python count_normal_words.py Job@549:db30e:25bd8: Status: [map] 0 waiting, 1 running, 0 done, 0 failed 2012/11/25 21:43:26 master New job initialized! 2012/11/25 21:43:26 master Starting job 2012/11/25 21:43:26 master Starting map phase 2012/11/25 21:43:26 master map:0 assigned to solice 2012/11/25 21:43:26 master ERROR: Job failed: Worker at 'solice' died: Traceback (most recent call last): File "/home/DISCO/data/solice/01/Job@549:db30e:25bd8/usr/local/lib/python2.7/site-packages/disco/worker/__init__.py", line 329, in main job.worker.start(task, job, **jobargs) File "/home/DISCO/data/solice/01/Job@549:db30e:25bd8/usr/local/lib/python2.7/site-packages/disco/worker/__init__.py", line 290, in start self.run(task, job, **jobargs) File "/home/DISCO/data/solice/01/Job@549:db30e:25bd8/usr/local/lib/python2.7/site-packages/disco/worker/classic/worker.py", line 286, in run getattr(self, task.mode)(task, params) File "/home/DISCO/data/solice/01/Job@549:db30e:25bd8/usr/local/lib/python2.7/site-packages/disco/worker/classic/worker.py", line 299, in map for key, val in self['map'](entry, params): File "count_normal_words.py", line 12, in map File "/usr/lib64/python2.7/json/__init__.py", line 326, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded 2012/11/25 21:43:26 master WARN: Job killed Status: [map] 1 waiting, 0 running, 0 done, 1 failed Traceback (most recent call last): File "count_normal_words.py", line 28, in <module> for word, count in result_iterator(job.wait(show=True)): File "/usr/local/lib/python2.7/site-packages/disco/core.py", line 348, in wait timeout, poll_interval * 1000) File "/usr/local/lib/python2.7/site-packages/disco/core.py", line 309, in check_results raise JobError(Job(name=jobname, master=self), "Status %s" % status) disco.error.JobError: Job Job@549:db30e:25bd8 failed: Status dead The Error states: ValueError: No JSON object could be decoded. Again, this works fine using the text file as input but now DDFS. Any ideas, I'm open to suggestions?

Read the article

Slowdowns when reading from an urlconnection's inputstream (even with byte[] and buffers)

- by user342677

Ok so after spending two days trying to figure out the problem, and reading about dizillion articles, i finally decided to man up and ask to for some advice(my first time here). Now to the issue at hand - I am writing a program which will parse api data from a game, namely battle logs. There will be A LOT of entries in the database(20+ million) and so the parsing speed for each battle log page matters quite a bit. The pages to be parsed look like this: http://api.erepublik.com/v1/feeds/battle_logs/10000/0. (see source code if using chrome, it doesnt display the page right). It has 1000 hit entries, followed by a little battle info(lastpage will have <1000 obviously). On average, a page contains 175000 characters, UTF-8 encoding, xml format(v 1.0). Program will run locally on a good PC, memory is virtually unlimited(so that creating byte[250000] is quite ok). The format never changes, which is quite convenient. Now, I started off as usual: //global vars,class declaration skipped public WebObject(String url_string, int connection_timeout, int read_timeout, boolean redirects_allowed, String user_agent) throws java.net.MalformedURLException, java.io.IOException { // Open a URL connection java.net.URL url = new java.net.URL(url_string); java.net.URLConnection uconn = url.openConnection(); if (!(uconn instanceof java.net.HttpURLConnection)) { throw new java.lang.IllegalArgumentException("URL protocol must be HTTP"); } conn = (java.net.HttpURLConnection) uconn; conn.setConnectTimeout(connection_timeout); conn.setReadTimeout(read_timeout); conn.setInstanceFollowRedirects(redirects_allowed); conn.setRequestProperty("User-agent", user_agent); } public void executeConnection() throws IOException { try { is = conn.getInputStream(); //global var l = conn.getContentLength(); //global var } catch (Exception e) { //handling code skipped } } //getContentStream and getLength methods which just return'is' and 'l' are skipped Here is where the fun part began. I ran some profiling (using System.currentTimeMillis()) to find out what takes long ,and what doesnt. The call to this method takes only 200ms on avg public InputStream getWebPageAsStream(int battle_id, int page) throws Exception { String url = "http://api.erepublik.com/v1/feeds/battle_logs/" + battle_id + "/" + page; WebObject wobj = new WebObject(url, 10000, 10000, true, "Mozilla/5.0 " + "(Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)"); wobj.executeConnection(); l = wobj.getContentLength(); // global variable return wobj.getContentStream(); //returns 'is' stream } 200ms is quite expected from a network operation, and i am fine with it. BUT when i parse the inputStream in any way(read it into string/use java XML parser/read it into another ByteArrayStream) the process takes over 1000ms! for example, this code takes 1000ms IF i pass the stream i got('is') above from getContentStream() directly to this method: public static Document convertToXML(InputStream is) throws ParserConfigurationException, IOException, SAXException { DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); DocumentBuilder db = dbf.newDocumentBuilder(); Document doc = db.parse(is); doc.getDocumentElement().normalize(); return doc; } this code too, takes around 920ms IF the initial InputStream 'is' is passed in(dont read into the code itself - it just extracts the data i need by directly counting the characters, which can be done thanks to the rigid api feed format): public static parsedBattlePage convertBattleToXMLWithoutDOM(InputStream is) throws IOException { // Point A BufferedReader br = new BufferedReader(new InputStreamReader(is)); LinkedList ll = new LinkedList(); String str = br.readLine(); while (str != null) { ll.add(str); str = br.readLine(); } if (((String) ll.get(1)).indexOf("error") != -1) { return new parsedBattlePage(null, null, true, -1); } //Point B Iterator it = ll.iterator(); it.next(); it.next(); it.next(); it.next(); String[][] hits_arr = new String[1000][4]; String t_str = (String) it.next(); String tmp = null; int j = 0; for (int i = 0; t_str.indexOf("time") != -1; i++) { hits_arr[i][0] = t_str.substring(12, t_str.length() - 11); tmp = (String) it.next(); hits_arr[i][1] = tmp.substring(14, tmp.length() - 9); tmp = (String) it.next(); hits_arr[i][2] = tmp.substring(15, tmp.length() - 10); tmp = (String) it.next(); hits_arr[i][3] = tmp.substring(18, tmp.length() - 13); it.next(); it.next(); t_str = (String) it.next(); j++; } String[] b_info_arr = new String[9]; int[] space_nums = {13, 10, 13, 11, 11, 12, 5, 10, 13}; for (int i = 0; i < space_nums.length; i++) { tmp = (String) it.next(); b_info_arr[i] = tmp.substring(space_nums[i] + 4, tmp.length() - space_nums[i] - 1); } //Point C return new parsedBattlePage(hits_arr, b_info_arr, false, j); } I have tried replacing the default BufferedReader with BufferedReader br = new BufferedReader(new InputStreamReader(is), 250000); This didnt change much. My second try was to replace the code between A and B with: Iterator it = IOUtils.lineIterator(is, "UTF-8"); Same result, except this time A-B was 0ms, and B-C was 1000ms, so then every call to it.next() must have been consuming some significant time.(IOUtils is from apache-commons-io library). And here is the culprit - the time taken to parse the stream to string, be it by an iterator or BufferedReader in ALL cases was about 1000ms, while the rest of the code took 0ms(e.g. irrelevant). This means that parsing the stream to LinkedList, or iterating over it, for some reason was eating up a lot of my system resources. question was - why? Is it just the way java is made...no...thats just stupid, so I did another experiment. In my main method I added after the getWebPageAsStream(): //Point A ba = new byte[l]; // 'l' comes from wobj.getContentLength above bytesRead = is.read(ba); //'is' is our URLConnection original InputStream offset = bytesRead; while (bytesRead != -1) { bytesRead = is.read(ba, offset - 1, l - offset); offset += bytesRead; } //Point B InputStream is2 = new ByteArrayInputStream(ba); //Now just working with 'is2' - the "copied" stream The InputStream-byte[] conversion took again 1000ms - this is the way many ppl suggested to read an InputStream, and stil it is slow. And guess what - the 2 parser methods above (convertToXML() and convertBattlePagetoXMLWithoutDOM(), when passed 'is2' instead of 'is' took, in all 4 cases, under 50ms to complete. I read a suggestion that the stream waits for connection to close before unblocking, so i tried using HttpComponentsClient 4.0 (http://hc.apache.org/httpcomponents-client/index.html) instead, but the initial InputStream took just as long to parse. e.g. this code: public InputStream getWebPageAsStream2(int battle_id, int page) throws Exception { String url = "http://api.erepublik.com/v1/feeds/battle_logs/" + battle_id + "/" + page; HttpClient httpclient = new DefaultHttpClient(); HttpGet httpget = new HttpGet(url); HttpParams p = new BasicHttpParams(); HttpConnectionParams.setSocketBufferSize(p, 250000); HttpConnectionParams.setStaleCheckingEnabled(p, false); HttpConnectionParams.setConnectionTimeout(p, 5000); httpget.setParams(p); HttpResponse response = httpclient.execute(httpget); HttpEntity entity = response.getEntity(); l = (int) entity.getContentLength(); return entity.getContent(); } took even longer to process(50ms more for just the network) and the stream parsing times remained the same. Obviously it can be instantiated so as to not create HttpClient and properties every time(faster network time), but the stream issue wont be affected by that. So we come to the center problem - why does the initial URLConnection InputStream(or HttpClient InputStream) take so long to process, while any stream of same size and content created locally is orders of magnitude faster? I mean, the initial response is already somewhere in RAM, and I cant see any good reasong why it is processed so slowly compared to when a same stream is just created from a byte[]. Considering I have to parse million of entries and thousands of pages like that, a total processing time of almost 1.5s/page seems WAY WAY too long. Any ideas? P.S. Please ask in any more code is required - the only thing I do after parsing is make a PreparedStatement and put the entries into JavaDB in packs of 1000+, and the perfomance is ok ~ 200ms/1000entries, prb could be optimized with more cache but I didnt look into it much.

Read the article

Array from HTML forms - Internal Server Error

- by user1392411

I try to make a requestform for our school. But if I transmit this form I get a "Internal Server Error". I searched but I found nothing like that. Also I don't get a only this error message "Internal Server Error", nothing more. Any ideas why? <!DOCTYPE html> <html> <head> <title></title> <meta charset="utf-8"> <meta name="description" content=""> <link rel="stylesheet" href="../css/normalize.css" /> <link rel="stylesheet" href="../css/bestellformular.css" />   </head> <body> <section id="head"> </section> <form action="../php/einlesen.php" method="post"> <section id="body"> <h1>Bestellung</h1> <table> <colgroup> <col width="10%"> <col width="80%"> <col width="10%"> </colgroup> <tr> <th> Artikel-Nr. </th> <th> Artikel </th> <th> Menge </th> </tr> <tr> <td> <input type="number" placeholder="Artikel-Nr." name="articelnr[]" required /> </td> <td> <input type="text" placeholder="Name des Artikels" name="articelname[]" required /> </td> <td> <input type="number" placeholder="Menge" name="quantity[]" required /> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> <tr> <td> <input type="number" name="articelnr[]" /> </td> <td> <input type="text" name="articelname[]" /> </td> <td> <input type="number" name="quantity[]"/> </td> </tr> </table> </section> <section id="info"> <div class="left"> <hr /> <p> Kundennummer <input type="text" /> </p> </div> <div class="right"> <table> <tr> <td>Firma</td> <td><input type="text" name="company"required /></td> </tr> <tr> <td>Ort, PLZ</td> <td><input type="text" name="place" required /></td> <td><input type="number" name="plz" class="number" required /></td> </tr> <tr> <td>Straße, Nr.</td> <td><input type="text" name="street"required /></td> <td><input type="number" name="streetnr" class="number" required /></td> </tr> <tr> <td>Telefon</td> <td><input type="tel" name="tel" required /></td> </tr> <tr> <td>Fax</td> <td><input type="text" name="fax" required /></td> </tr> <tr> <td>E-Mail</td> <td><input type="email" name="email" required /></td> </tr> <tr> <td>Datum</td> <td><input type="date" name="date" required placeholder="tt.mm.jj"/></td> </tr> </table> </div> </section> <section id="submit"> <input type="checkbox" name="agb" required /> Ich habe die <a href="../docs/agb.pdf">AGB</a> gelesen und akzeptiere diese. <input type="submit" value="Bestellung Abschicken"/> </section> </form> <section id="footer"> <hr /> </section> </body> </html> Any Ideas why? The data is sent to a yet empty PHP document. The bracelets in the name tag are used to get an array. My PHP version is 5.3.8

Developer IT