Search Results

Search found 196 results on 8 pages for 'getresponsestream'.

Page 8/8 | < Previous Page | 4 5 6 7 8 

  • Why does IHttpAsyncHandler leak memory under load?

    - by Anton
    I have noticed that the .NET IHttpAsyncHandler (and the IHttpHandler, to a lesser degree) leak memory when subjected to concurrent web requests. In my tests, the development web server (Cassini) jumps from 6MB memory to over 100MB, and once the test is finished, none of it is reclaimed. The problem can be reproduced easily. Create a new solution (LeakyHandler) with two projects: An ASP.NET web application (LeakyHandler.WebApp) A Console application (LeakyHandler.ConsoleApp) In LeakyHandler.WebApp: Create a class called TestHandler that implements IHttpAsyncHandler. In the request processing, do a brief Sleep and end the response. Add the HTTP handler to Web.config as test.ashx. In LeakyHandler.ConsoleApp: Generate a large number of HttpWebRequests to test.ashx and execute them asynchronously. As the number of HttpWebRequests (sampleSize) is increased, the memory leak is made more and more apparent. LeakyHandler.WebApp TestHandler.cs namespace LeakyHandler.WebApp { public class TestHandler : IHttpAsyncHandler { #region IHttpAsyncHandler Members private ProcessRequestDelegate Delegate { get; set; } public delegate void ProcessRequestDelegate(HttpContext context); public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData) { Delegate = ProcessRequest; return Delegate.BeginInvoke(context, cb, extraData); } public void EndProcessRequest(IAsyncResult result) { Delegate.EndInvoke(result); } #endregion #region IHttpHandler Members public bool IsReusable { get { return true; } } public void ProcessRequest(HttpContext context) { Thread.Sleep(10); context.Response.End(); } #endregion } } LeakyHandler.WebApp Web.config <?xml version="1.0"?> <configuration> <system.web> <compilation debug="false" /> <httpHandlers> <add verb="POST" path="test.ashx" type="LeakyHandler.WebApp.TestHandler" /> </httpHandlers> </system.web> </configuration> LeakyHandler.ConsoleApp Program.cs namespace LeakyHandler.ConsoleApp { class Program { private static int sampleSize = 10000; private static int startedCount = 0; private static int completedCount = 0; static void Main(string[] args) { Console.WriteLine("Press any key to start."); Console.ReadKey(); string url = "http://localhost:3000/test.ashx"; for (int i = 0; i < sampleSize; i++) { HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url); request.Method = "POST"; request.BeginGetResponse(GetResponseCallback, request); Console.WriteLine("S: " + Interlocked.Increment(ref startedCount)); } Console.ReadKey(); } static void GetResponseCallback(IAsyncResult result) { HttpWebRequest request = (HttpWebRequest)result.AsyncState; HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result); try { using (Stream stream = response.GetResponseStream()) { using (StreamReader streamReader = new StreamReader(stream)) { streamReader.ReadToEnd(); System.Console.WriteLine("C: " + Interlocked.Increment(ref completedCount)); } } response.Close(); } catch (Exception ex) { System.Console.WriteLine("Error processing response: " + ex.Message); } } } }

    Read the article

  • KeepAliveException when using HttpWebRequest.GetResponse

    - by Lucas
    I am trying to POST an attachment to CouchDB using the HttpWebRequest. However, when I attempt "response = (HttpWebResponse)httpWebRequest.GetResponse();" I receive a WebException with the message "The underlying connection was closed: A connection that was expected to be kept alive was closed by the server." I have found some articles stating that setting the keepalive to false and httpversion to 1.0 resolves the situation. I am finding that it does not yeilding the exact same error, plus I do not want to take that approach as I do not want to use the 1.0 version due to how it handles the connection. Any suggestions or ideas are welcome. I'll try them all until one works! public ServerResponse PostAttachment(Server server, Database db, Attachment attachment) { Stream dataStream; HttpWebResponse response = null; StreamReader sr = null; byte[] buffer; string json; string boundary = "----------------------------" + DateTime.Now.Ticks.ToString("x"); string headerTemplate = "Content-Disposition: form-data; name=\"_attachments\"; filename=\"" + attachment.Filename + "\"\r\n Content-Type: application/octet-stream\r\n\r\n"; byte[] headerbytes = System.Text.Encoding.UTF8.GetBytes(headerTemplate); byte[] boundarybytes = System.Text.Encoding.ASCII.GetBytes("\r\n--" + boundary + "\r\n"); HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("http://" + server.Host + ":" + server.Port.ToString() + "/" + db.Name + "/" + attachment.Document.Id); httpWebRequest.ContentType = "multipart/form-data; boundary=" + boundary; httpWebRequest.Method = "POST"; httpWebRequest.KeepAlive = true; httpWebRequest.ContentLength = attachment.Stream.Length + headerbytes.Length + boundarybytes.Length; if (!string.IsNullOrEmpty(server.EncodedCredentials)) httpWebRequest.Headers.Add("Authorization", server.EncodedCredentials); if (!attachment.Stream.CanRead) throw new System.NotSupportedException("The stream cannot be read."); // Get the request stream try { dataStream = httpWebRequest.GetRequestStream(); } catch (Exception e) { throw new WebException("Failed to get the request stream.", e); } buffer = new byte[server.BufferSize]; int bytesRead; dataStream.Write(headerbytes,0,headerbytes.Length); attachment.Stream.Position = 0; while ((bytesRead = attachment.Stream.Read(buffer, 0, buffer.Length)) > 0) { dataStream.Write(buffer, 0, bytesRead); } dataStream.Write(boundarybytes, 0, boundarybytes.Length); dataStream.Close(); // send the request and get the response try { response = (HttpWebResponse)httpWebRequest.GetResponse(); } catch (Exception e) { throw new WebException("Invalid response received from server.", e); } // get the server's response json try { dataStream = response.GetResponseStream(); sr = new StreamReader(dataStream); json = sr.ReadToEnd(); } catch (Exception e) { throw new WebException("Failed to access the response stream.", e); } // close up all our streams and response sr.Close(); dataStream.Close(); response.Close(); // Deserialize the server response return ConvertTo.JsonToServerResponse(json); }

    Read the article

  • HttpWebRequest: How to find a postal code at Canada Post through a WebRequest with x-www-form-enclos

    - by Will Marcouiller
    I'm currently writing some tests so that I may improve my skills with the Internet interaction through Windows Forms. One of those tests is to find a postal code which should be returned by Canada Post website. My default URL setting is set to: http://www.canadapost.ca/cpotools/apps/fpc/personal/findByCity?execution=e4s1 The required form fields are: streetNumber, streetName, city, province The contentType is "application/x-www-form-enclosed" EDIT: Please consider the value "application/x-www-form-encoded" instead of point 3 value as the contentType. (Thanks EricLaw-MSFT!) The result I get is not the result expected. I get the HTML source code of the page where I could manually enter the information to find the postal code, but not the HTML source code with the found postal code. Any idea of what I'm doing wrong? Shall I consider going the XML way? Is it first of all possible to search on Canada Post anonymously? Here's a code sample for better description: public static string FindPostalCode(ICanadadianAddress address) { var postData = string.Concat(string.Format("&streetNumber={0}", address.StreetNumber) , string.Format("&streetName={0}", address.StreetName) , string.Format("&city={0}", address.City) , string.Format("&province={0}", address.Province)); var encoding = new ASCIIEncoding(); byte[] postDataBytes = encoding.GetBytes(postData); request = (HttpWebRequest)WebRequest.Create(DefaultUrlSettings); request.ImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Anonymous; request.Container = new CookieContainer(); request.Timeout = 10000; request.ContentType = contentType; request.ContentLength = postDataBytes.LongLength; request.Method = @"post"; var senderStream = new StreamWriter(request.GetRequestStream()); senderStream.Write(postDataBytes, 0, postDataBytes.Length); senderStream.Close(); string htmlResponse = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd(); return processedResult(htmlResponse); // Processing the HTML source code parsing, etc. } I seem stuck in a bottle neck in my point of view. I find no way out to the desired result. EDIT: There seems to have to parameters as for the ContentType of this site. Let me explain. There's one with the "meta"-variables which stipulates the following: meta http-equiv="Content-Type" content="application/xhtml+xml, text/xml, text/html; charset=utf-8" And another one later down the code that is read as: form id="fpcByAdvancedSearch:fpcSearch" name="fpcByAdvancedSearch:fpcSearch" method="post" action="/cpotools/apps/fpc/personal/findByCity?execution=e1s1" enctype="application/x-www-form-urlencoded" My question is the following: With which one do I have to stick? Let me guess, the first ContentType is to be considered as the second is only for another request to a function or so when the data is posted? EDIT: As per request, the closer to the solution I am is listed under this question: WebRequest: How to find a postal code using a WebRequest against this ContentType=”application/xhtml+xml, text/xml, text/html; charset=utf-8”? Thanks for any help! :-)

    Read the article

  • Receicing POST data in ASP.NET

    - by grast
    Hi, I want to use ASP for code generation in a C# desktop application. To achieve this, I set up a simple host (derived from System.MarshalByRefObject) that processes a System.Web.Hosting.SimpleWorkerRequest via HttpRuntime.ProcessRequest. This processes the ASPX script specified by the incoming request (using System.Net.HttpListener to wait for requests). The client-part is represented by a System.ComponentModel.BackgroundWorker that builds the System.Net.HttpWebRequest and receives the response from the server. A simplified version of my client-part-code looks like this: private void SendRequest(object sender, DoWorkEventArgs e) { // create request with GET parameter var uri = "http://localhost:9876/test.aspx?getTest=321"; var request = (HttpWebRequest)WebRequest.Create(uri); // append POST parameter request.Method = "POST"; request.ContentType = "application/x-www-form-urlencoded"; var postData = Encoding.Default.GetBytes("postTest=654"); var postDataStream = request.GetRequestStream(); postDataStream.Write(postData, 0, postData.Length); // send request, wait for response and store/print content using (var response = (HttpWebResponse)request.GetResponse()) { using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8)) { _processsedContent = reader.ReadToEnd(); Debug.Print(_processsedContent); } } } My server-part-code looks like this (without exception-handling etc.): public void ProcessRequests() { // HttpListener at http://localhost:9876/ var listener = SetupListener(); // SimpleHost created by ApplicationHost.CreateApplicationHost var host = SetupHost(); while (_running) { var context = listener.GetContext(); using (var writer = new StreamWriter(context.Response.OutputStream)) { // process ASP script and send response back to client host.ProcessRequest(GetPage(context), GetQuery(context), writer); } context.Response.Close(); } } So far all this works fine as long as I just use GET parameters. But when it comes to receiving POST data in my ASPX script I run into trouble. For testing I use the following script: // GET parameters are working: var getTest = Request.QueryString["getTest"]; Response.Write("getTest: " + getTest); // prints "getTest: 321" // don't know how to access POST parameters: var postTest1 = Request.Form["postTest"]; // Request.Form is empty?! Response.Write("postTest1: " + postTest1); // so this prints "postTest1: " var postTest2 = Request.Params["postTest"]; // Request.Params is empty?! Response.Write("postTest2: " + postTest2); // so this prints "postTest2: " It seems that the System.Web.HttpRequest object I'm dealing with in ASP does not contain any information about my POST parameter "postTest". I inspected it in debug mode and none of the members did contain neither the parameter-name "postTest" nor the parameter-value "654". I also tried the BinaryRead method of Request, but unfortunately it is empty. This corresponds to Request.InputStream==null and Request.ContentLength==0. And to make things really confusing the Request.HttpMethod member is set to "GET"?! To isolate the problem I tested the code by using a PHP script instead of the ASPX script. This is very simple: print_r($_GET); // prints all GET variables print_r($_POST); // prints all POST variables And the result is: Array ( [getTest] = 321 ) Array ( [postTest] = 654 ) So with the PHP script it works, I can access the POST data. Why does the ASPX script don't? What am I doing wrong? Is there a special accessor or method in the Response object? Can anyone give a hint or even know how to solve this? Thanks in advance.

    Read the article

  • Help deciphering exception details from WebRequestCreator when setting ContentType to "application/json"

    - by Stephen Patten
    Hello, This one is real simple, run this Silverlight4 example with the ContentType property commented out and you'll get back a response from from my service in xml. Now uncomment the property and run it and you'll get an exception similar to this one. System.Net.ProtocolViolationException occurred Message=A request with this method cannot have a request body. StackTrace: at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state) at System.Net.Browser.ClientHttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at com.patten.silverlight.ViewModels.WebRequestLiteViewModel.<MakeCall>b__0(IAsyncResult cb) InnerException: What I am trying to accomplish is just pulling down some JSON formatted data from my wcf endpoint. Can this really be this hard, or is it another classic example of just overlooking something simple. Edit: While perusing SO, I noticed similar posts, like this one Why am I getting ProtocolViolationException when trying to use HttpWebRequest? Thank you, Stephen try { Address = "http://stephenpattenconsulting.com/Services/GetFoodDescriptionsLookup(2100)"; // Get the URI Uri httpSite = new Uri(Address); // Create the request object using the Browsers networking stack // HttpWebRequest wreq = (HttpWebRequest)WebRequest.Create(httpSite); // Create the request using the operating system's networking stack HttpWebRequest wreq = (HttpWebRequest)WebRequestCreator.ClientHttp.Create(httpSite); // http://stackoverflow.com/questions/239725/c-webrequest-class-and-headers // These headers have been set, so use the property that has been exposed to change them // wreq.Headers[HttpRequestHeader.ContentType] = "application/json"; //wreq.ContentType = "application/json"; // Issue the async request. // http://timheuer.com/blog/archive/2010/04/23/silverlight-authorization-header-access.aspx wreq.BeginGetResponse((cb) => { HttpWebRequest rq = cb.AsyncState as HttpWebRequest; HttpWebResponse resp = rq.EndGetResponse(cb) as HttpWebResponse; StreamReader rdr = new StreamReader(resp.GetResponseStream()); string result = rdr.ReadToEnd(); Jounce.Framework.JounceHelper.ExecuteOnUI(() => { Result = result; }); rdr.Close(); }, wreq); } catch (WebException ex) { Jounce.Framework.JounceHelper.ExecuteOnUI(() => { Error = ex.Message; }); } catch (Exception ex) { Jounce.Framework.JounceHelper.ExecuteOnUI(() => { Error = ex.Message; }); } EDIT: This is how the WCF 4 end point is configured, primarily 'adapted' from this link http://geekswithblogs.net/michelotti/archive/2010/08/21/restful-wcf-services-with-no-svc-file-and-no-config.aspx [ServiceContract] public interface IRDA { [OperationContract] IList<FoodDescriptionLookup> GetFoodDescriptionsLookup(String id); [OperationContract] FOOD_DES GetFoodDescription(String id); [OperationContract] FOOD_DES InsertFoodDescription(FOOD_DES foodDescription); [OperationContract] FOOD_DES UpdateFoodDescription(String id, FOOD_DES foodDescription); [OperationContract] void DeleteFoodDescription(String id); } // RESTfull service [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)] public class RDAService : IRDA { [WebGet(UriTemplate = "FoodDescription({id})")] public FOOD_DES GetFoodDescription(String id) { ... } [AspNetCacheProfile("GetFoodDescriptionsLookup")] [WebGet(UriTemplate = "GetFoodDescriptionsLookup({id})")] public IList<FoodDescriptionLookup> GetFoodDescriptionsLookup(String id) { return rda.GetFoodDescriptionsLookup(id); ; } [WebInvoke(UriTemplate = "FoodDescription", Method = "POST")] public FOOD_DES InsertFoodDescription(FOOD_DES foodDescription) { ... } [WebInvoke(UriTemplate = "FoodDescription({id})", Method = "PUT")] public FOOD_DES UpdateFoodDescription(String id, FOOD_DES foodDescription) { ... } [WebInvoke(UriTemplate = "FoodDescription({id})", Method = "DELETE")] public void DeleteFoodDescription(String id) { ... } } And the portion of my web.config that pertains to WCF <system.serviceModel> <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" /> <standardEndpoints> <webHttpEndpoint> <standardEndpoint name="" helpEnabled="true" automaticFormatSelectionEnabled="true" /> </webHttpEndpoint> </standardEndpoints> </system.serviceModel>

    Read the article

  • Receiving POST data in ASP.NET

    - by grast
    Hi, I want to use ASP for code generation in a C# desktop application. To achieve this, I set up a simple host (derived from System.MarshalByRefObject) that processes a System.Web.Hosting.SimpleWorkerRequest via HttpRuntime.ProcessRequest. This processes the ASPX script specified by the incoming request (using System.Net.HttpListener to wait for requests). The client-part is represented by a System.ComponentModel.BackgroundWorker that builds the System.Net.HttpWebRequest and receives the response from the server. A simplified version of my client-part-code looks like this: private void SendRequest(object sender, DoWorkEventArgs e) { // create request with GET parameter var uri = "http://localhost:9876/test.aspx?getTest=321"; var request = (HttpWebRequest)WebRequest.Create(uri); // append POST parameter request.Method = "POST"; request.ContentType = "application/x-www-form-urlencoded"; var postData = Encoding.Default.GetBytes("postTest=654"); var postDataStream = request.GetRequestStream(); postDataStream.Write(postData, 0, postData.Length); // send request, wait for response and store/print content using (var response = (HttpWebResponse)request.GetResponse()) { using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8)) { _processsedContent = reader.ReadToEnd(); Debug.Print(_processsedContent); } } } My server-part-code looks like this (without exception-handling etc.): public void ProcessRequests() { // HttpListener at http://localhost:9876/ var listener = SetupListener(); // SimpleHost created by ApplicationHost.CreateApplicationHost var host = SetupHost(); while (_running) { var context = listener.GetContext(); using (var writer = new StreamWriter(context.Response.OutputStream)) { // process ASP script and send response back to client host.ProcessRequest(GetPage(context), GetQuery(context), writer); } context.Response.Close(); } } So far all this works fine as long as I just use GET parameters. But when it comes to receiving POST data in my ASPX script I run into trouble. For testing I use the following script: // GET parameters are working: var getTest = Request.QueryString["getTest"]; Response.Write("getTest: " + getTest); // prints "getTest: 321" // don't know how to access POST parameters: var postTest1 = Request.Form["postTest"]; // Request.Form is empty?! Response.Write("postTest1: " + postTest1); // so this prints "postTest1: " var postTest2 = Request.Params["postTest"]; // Request.Params is empty?! Response.Write("postTest2: " + postTest2); // so this prints "postTest2: " It seems that the System.Web.HttpRequest object I'm dealing with in ASP does not contain any information about my POST parameter "postTest". I inspected it in debug mode and none of the members did contain neither the parameter-name "postTest" nor the parameter-value "654". I also tried the BinaryRead method of Request, but unfortunately it is empty. This corresponds to Request.InputStream==null and Request.ContentLength==0. And to make things really confusing the Request.HttpMethod member is set to "GET"?! To isolate the problem I tested the code by using a PHP script instead of the ASPX script. This is very simple: print_r($_GET); // prints all GET variables print_r($_POST); // prints all POST variables And the result is: Array ( [getTest] = 321 ) Array ( [postTest] = 654 ) So with the PHP script it works, I can access the POST data. Why does the ASPX script don't? What am I doing wrong? Is there a special accessor or method in the Response object? Can anyone give a hint or even know how to solve this? Thanks in advance.

    Read the article

  • HttpWebRequest response produces HTTP 422. Why?

    - by Simon
    Hi there. I'm trying to programmatically send a POST-request to a web-server in order to login an then perform other requests that require a login. This is my code: Encoding.UTF8.GetBytes( String.Format( "login={0}&password={1}&authenticity_token={2}", username, password, token)); //Create HTTP-request for login HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://www.xxx.xx/xx/xx"); request.Method = "POST"; request.ContentType = "application/x-www-form-urlencoded"; request.ContentLength = data.Length; request.CookieContainer = new CookieContainer(); request.Accept = "application/xml,application/xhtml+xml,text/html; +"q=0.9,text/plain ;q=0.8,image/png,*/*;q=0.5"; request.Referer = "http://www.garzantilinguistica.it/it/session"; request.Headers.Add("Accept-Language", "de-DE"); request.Headers.Add("Origin", "http://www.xxx.xx"); request.UserAgent = "C#"; request.Headers.Add("Accept-Encoding", "gzip, deflate"); After sending the request //Send post request var requestStream = request.GetRequestStream(); requestStream.Write(data, 0, data.Length); requestStream.Flush(); requestStream.Close(); ... I want to get the servers response: //Get Response StreamReader responseStreamReader = new StreamReader( request.GetResponse().GetResponseStream()); //WebException: HTTP 422! string content = responseStreamReader.ReadToEnd(); This piece of code fires the WebException, that tells me the server responded with HTTP 422 (unprocessable entity due to semantic errors) Then I compared (using a TCP/IP sniffers) the requests of my program and the browser (which of course produces a valid POST-request and gets the right response). (1) My program's request: POST /it/session HTTP/1.1 Content-Type: application/x-www-form-urlencoded Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain; q=0.8,image/png,*/*;q=0.5 Referer: http://www.garzantilinguistica.it/it/session Accept-Language: de-DE Origin: http://www.garzantilinguistica.it User-Agent: C# Accept-Encoding: gzip, deflate Host: www.garzantilinguistica.it Content-Length: 111 Expect: 100-continue Connection: Keep-Alive HTTP/1.1 100 Continue [email protected]&password=machivui&authenticity_token=4vLgtwP3nFNg4NeuG4MbUnU7sy4z91Wi8WJXH0POFmg= HTTP/1.1 422 Unprocessable Entity (2) The browser's request: POST /it/session HTTP/1.1 Host: www.garzantilinguistica.it Referer: http://www.garzantilinguistica.it/it/session Accept: application/xml,application/xhtml+xml,text/html;q=0.9, text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: de-DE Origin: http://www.garzantilinguistica.it User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7 Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded Cookie: __utma=244184339.652523587.1275208707.1275208707.1275211298.2; __utmb=244184339.20.10.1275211298; __utmc=244184339; __utmz=244184339.1275208707.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _garzanti2009_session=BAh7CDoPc2Vzc2lvbl9pZCIlZDg4MWZjNjg2YTRhZWE0NDQ0ZTJmMTU2YWY4ZTQ1NGU6EF9jc3JmX3Rva2VuIjFqRWdLdll3dTYwOTVVTEpNZkt6dG9jUCtaZ0o4V0FnV2V5ZnpuREx6QUlZPSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNoSGFzaHsGOgplcnJvciIVbG9naW4gbm9uIHZhbGlkbwY6CkB1c2VkewY7CFQ%3D--4200fa769898dd156faa49e457baf660cf068d08 Content-Length: 144 Connection: keep-alive authenticity_token=jEgKvYwu6095ULJMfKztocP%2BZgJ8WAgWeyfznDLzAIY%3D&login=thespider14%40hotmail.com&password=machivui&remember_me=1&commit=Entra HTTP/1.1 302 Found Can someone help to understand which part of the request I am missing or what the main difference between the browser's and my request is? Why am I getting that 422?

    Read the article

  • String from Httpresponse not passing full value.

    - by Shekhar_Pro
    HI i am in desperate need for help here, I am making a web request and getting a json string with Response.ContentLenth=2246 but when i parse it in a string it gives only few 100 characters, i traked it down that it is only getting values less than 964. strings length is still 2246 but remaining values are just (\0) null characters. Its also giving an error Unterminated string passed in. (2246): at following line FacebookFeed feed = sr.Deserialize<FacebookFeed>(data); It works fine if the response stream contains characters less than 964 chars. Following is the extract from the full code error encountered in last line. System.Web.Script.Serialization.JavaScriptSerializer sr = new System.Web.Script.Serialization.JavaScriptSerializer(); System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(@"https://graph.facebook.com/100000570310973_181080451920964"); req.Method = "GET"; System.Net.HttpWebResponse res = (System.Net.HttpWebResponse)req.GetResponse(); byte[] resp = new byte[(int)res.ContentLength]; res.GetResponseStream().Read(resp, 0, (int)res.ContentLength); string data = Encoding.UTF8.GetString(resp); FacebookFeed feed = sr.Deserialize<FacebookFeed>(data); error given is Unterminated string passed in. (2246): {"id":"100000570310973_1810804519209........ (with rest of data in the string data including null chars) following is the shape of classes used in my code: public class FacebookFeed { public string id { get; set; } public NameIdPair from { get; set; } public NameIdPair to { get; set; } public string message { get; set; } public Uri link{get;set;} public string name{get; set;} public string caption { get; set; } public Uri icon { get; set; } public NameLinkPair[] actions { get; set; } public string type { get; set; } public NameIdPair application { get; set; } //Mentioned in Graph API as attribution public DateTime created_time { get; set; } public DateTime updated_time { get; set; } public FacebookPostLikes likes { get; set; } } public class NameIdPair { public string name { get; set; } public string id { get; set; } } public class NameLinkPair { public string name { get; set; } public Uri link{get; set;} } public class FacebookPostLikes { public NameIdPair[] data { get; set; } public int count { get; set; } }

    Read the article

  • Why does sending post data with WebRequest take so long?

    - by Paramiliar
    I am currently creating a C# application to tie into a php / MySQL online system. The application needs to send post data to scripts and get the response. When I send the following data username=test&password=test I get the following responses... Starting request at 22/04/2010 12:15:42 Finished creating request : took 00:00:00.0570057 Transmitting data at 22/04/2010 12:15:42 Transmitted the data : took 00:00:06.9316931 <<-- Getting the response at 22/04/2010 12:15:49 Getting response 00:00:00.0360036 Finished response 00:00:00.0360036 Entire call took 00:00:07.0247024 As you can see it is taking 6 seconds to actually send the data to the script, I have done further testing bye sending data from telnet and by sending post data from a local file to the url and they dont even take a second so this is not a problem with the hosted script on the site. Why is it taking 6 seconds to transmit the data when it is two simple strings? I use a custom class to send the data class httppostdata { WebRequest request; WebResponse response; public string senddata(string url, string postdata) { var start = DateTime.Now; Console.WriteLine("Starting request at " + start.ToString()); // create the request to the url passed in the paramaters request = (WebRequest)WebRequest.Create(url); // set the method to post request.Method = "POST"; // set the content type and the content length request.ContentType = "application/x-www-form-urlencoded"; request.ContentLength = postdata.Length; // convert the post data into a byte array byte[] byteData = Encoding.UTF8.GetBytes(postdata); var end1 = DateTime.Now; Console.WriteLine("Finished creating request : took " + (end1 - start)); var start2 = DateTime.Now; Console.WriteLine("Transmitting data at " + start2.ToString()); // get the request stream and write the data to it Stream dataStream = request.GetRequestStream(); dataStream.Write(byteData, 0, byteData.Length); dataStream.Close(); var end2 = DateTime.Now; Console.WriteLine("Transmitted the data : took " + (end2 - start2)); // get the response var start3 = DateTime.Now; Console.WriteLine("Getting the response at " + start3.ToString()); response = request.GetResponse(); //Console.WriteLine(((WebResponse)response).StatusDescription); dataStream = response.GetResponseStream(); StreamReader reader = new StreamReader(dataStream); var end3 = DateTime.Now; Console.WriteLine("Getting response " + (end3 - start3)); // read the response string serverresponse = reader.ReadToEnd(); var end3a = DateTime.Now; Console.WriteLine("Finished response " + (end3a - start3)); Console.WriteLine("Entire call took " + (end3a - start)); //Console.WriteLine(serverresponse); reader.Close(); dataStream.Close(); response.Close(); return serverresponse; } } And to call it I use private void btnLogin_Click(object sender, EventArgs e) { // string postdata; if (txtUsername.Text.Length < 3 || txtPassword.Text.Length < 3) { MessageBox.Show("Missing your username or password."); } else { string postdata = "username=" + txtUsername.Text + "&password=" + txtPassword.Text; httppostdata myPost = new httppostdata(); string response = myPost.senddata("http://www.domainname.com/scriptname.php", postdata); MessageBox.Show(response); } }

    Read the article

  • Visual Studio reports that not all code path return a value, even though they do

    - by chris12892
    I have an API in NETMF C# that I am writing that includes a function to send an HTTP request. For those who are familiar with NETMF, this is a heavily modified version of the "webClient" example, which a simple application that demonstrates how to submit an HTTP request, and recive a response. In the sample, it simply prints the response and returns void,. In my version, however, I need it to return the HTTP response. For some reason, Visual Studio reports that not all code paths return a value, even though, as far as I can tell, they do. Here is my code... /// <summary> /// This is a modified webClient /// </summary> /// <param name="url"></param> private string httpRequest(string url) { // Create an HTTP Web request. HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest; // Set request.KeepAlive to use a persistent connection. request.KeepAlive = true; // Get a response from the server. WebResponse resp = request.GetResponse(); // Get the network response stream to read the page data. if (resp != null) { Stream respStream = resp.GetResponseStream(); string page = ""; byte[] byteData = new byte[4096]; char[] charData = new char[4096]; int bytesRead = 0; Decoder UTF8decoder = System.Text.Encoding.UTF8.GetDecoder(); int totalBytes = 0; // allow 5 seconds for reading the stream respStream.ReadTimeout = 5000; // If we know the content length, read exactly that amount of // data; otherwise, read until there is nothing left to read. if (resp.ContentLength != -1) { for (int dataRem = (int)resp.ContentLength; dataRem > 0; ) { Thread.Sleep(500); bytesRead = respStream.Read(byteData, 0, byteData.Length); if (bytesRead == 0) throw new Exception("Data laes than expected"); dataRem -= bytesRead; // Convert from bytes to chars, and add to the page // string. int byteUsed, charUsed; bool completed = false; totalBytes += bytesRead; UTF8decoder.Convert(byteData, 0, bytesRead, charData, 0, bytesRead, true, out byteUsed, out charUsed, out completed); page = page + new String(charData, 0, charUsed); } page = new String(System.Text.Encoding.UTF8.GetChars(byteData)); } else throw new Exception("No content-Length reported"); // Close the response stream. For Keep-Alive streams, the // stream will remain open and will be pushed into the unused // stream list. resp.Close(); return page; } } Any ideas? Thanks...

    Read the article

  • SOAP Service Request C#

    - by user3728352
    I have this code that tries to send a request to a soap server, I'm new to soap so i am not sure if the terms i am using are correct or not please correct me I am wrong. Basically i am accessing a web service method named getUserDomain via soap request Here is the code: public void CallWebService() { var _url = "https://....com/QcXmlWebService/QcXmlWebService.asmx?wsdl"; var _action = "https://....com/QcXmlWebService/QcXmlWebService.asmx?op=GetUserDomains"; XmlDocument soapEnvelopeXml = CreateSoapEnvelope(); HttpWebRequest webRequest = CreateWebRequest(_url, _action); InsertSoapEnvelopeIntoWebRequest(soapEnvelopeXml, webRequest); webRequest.BeginGetResponse(null, null); // begin async call to web request. IAsyncResult asyncResult = webRequest.BeginGetResponse(null, null); // suspend this thread until call is complete. You might want to // do something usefull here like update your UI. asyncResult.AsyncWaitHandle.WaitOne(); // get the response from the completed web request. string soapResult; using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult)) { using (StreamReader rd = new StreamReader(webResponse.GetResponseStream())) { soapResult = rd.ReadToEnd(); } Console.Write(soapResult); } } private HttpWebRequest CreateWebRequest(string url, string action) { HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url); webRequest.Headers.Add("SOAPAction", action); webRequest.ContentType = "text/xml;charset=\"utf-8\""; webRequest.Accept = "text/xml"; webRequest.Method = "POST"; return webRequest; } private XmlDocument CreateSoapEnvelope() { XmlDocument soapEnvelop = new XmlDocument(); string oRequest = ""; oRequest = @"<soap:Envelope xmlns:soap=""http://www.w3.org/2003/05/soap-envelope"" xmlns:qcx=""http://smething.com/QCXML"">"; oRequest = oRequest + "<soap:Header/>"; oRequest = oRequest + "<soap:Body>"; oRequest = oRequest + "<qcx:GetUserDomains>"; oRequest = oRequest + "<qcx:inputXml><![CDATA["; oRequest = oRequest + "<GetUserDomains>"; oRequest = oRequest + "<login>"; oRequest = oRequest + "<domain_name>MBB_BTS</domain_name>"; oRequest = oRequest + "<project_name>WCDMA_BTS_IV</project_name>"; oRequest = oRequest + "<user_name>user</user_name>"; oRequest = oRequest + "<password>pass</password>"; oRequest = oRequest + "</login>"; oRequest = oRequest + "</GetUserDomains>"; oRequest = oRequest + " ]]>"; oRequest = oRequest + "</qcx:inputXml>"; oRequest = oRequest + "</qcx:GetUserDomains>"; oRequest = oRequest + "</soap:Body>"; oRequest = oRequest + "</soap:Envelope>"; soapEnvelop.LoadXml(oRequest); return soapEnvelop; } private void InsertSoapEnvelopeIntoWebRequest(XmlDocument soapEnvelopeXml, HttpWebRequest webRequest) { using (Stream stream = webRequest.GetRequestStream()) { soapEnvelopeXml.Save(stream); } } This code i have seen somewhere in stack overflow before as an answer but i couldn't get it to work... The error im getting is threw exception System.net.webexception. the remote server returned an error :(500) internal server Thanks

    Read the article

  • Windows Phone 7 development: reading RSS feeds

    - by DigiMortal
    One limitation on Windows Phone 7 is related to System.Net namespace classes. There is no convenient way to read data from web. There is no WebClient class. There is no GetResponse() method – we have to do it all asynchronously because compact framework has limited set of classes we can use in our applications to communicate with internet. In this posting I will show you how to read RSS-feeds on Windows Phone 7. NB! This is my draft code and it may contain some design flaws and some questionable solutions. This code is intended to use as test-drive for Windows Phone 7 CTP developer tools and I don’t suppose you are going to use this code in production environment. Current state of my RSS-reader Currently my RSS-reader for Windows Phone 7 is very simple, primitive and uses almost all defaults that come out-of-box with Windows Phone 7 CTP developer tools. My first goal before going on with nicer user interface design was making RSS-reading work because instead of convenient classes from .NET Framework we have to use very limited classes from .NET Framework CE. This is why I took the reading of RSS-feeds as my first task. There are currently more things to solve regarding user-interface. As I am pretty new to all this Silverlight stuff I am not very sure if I can modify default controls easily or should I write my own controls that have better look and that work faster. The image on right shows you how my RSS-reader looks like right now. Upper side of screen is filled with list that shows headlines from this blog. The bottom part of screen is used to show description of selected posting. You can click on the image to see it in original size. In my next posting I will show you some improvements of my RSS-reader user interface that make it look nicer. But currently it is nice enough to make sure that RSS-feeds are read correctly. FeedItem class As this is most straight-forward part of the following code I will show you RSS-feed items class first. I think we have to stop on it because it is simple one. public class FeedItem {     public string Title { get; set; }     public string Description { get; set; }     public DateTime PublishDate { get; set; }     public List<string> Categories { get; set; }     public string Link { get; set; }       public FeedItem()     {         Categories = new List<string>();     } } RssClient RssClient takes feed URL and when asked it loads all items from feed and gives them back to caller through ItemsReceived event. Why it works this way? Because we can make responses only using asynchronous methods. I will show you in next section how to use this class. Although the code here is not very good but it works like expected. I will refactor this code later because it needs some more efforts and investigating. But let’s hope I find excellent solution. :) public class RssClient {     private readonly string _rssUrl;       public delegate void ItemsReceivedDelegate(RssClient client, IList<FeedItem> items);     public event ItemsReceivedDelegate ItemsReceived;       public RssClient(string rssUrl)     {         _rssUrl = rssUrl;     }       public void LoadItems()     {         var request = (HttpWebRequest)WebRequest.Create(_rssUrl);         var result = (IAsyncResult)request.BeginGetResponse(ResponseCallback, request);     }       void ResponseCallback(IAsyncResult result)     {         var request = (HttpWebRequest)result.AsyncState;         var response = request.EndGetResponse(result);           var stream = response.GetResponseStream();         var reader = XmlReader.Create(stream);         var items = new List<FeedItem>(50);           FeedItem item = null;         var pointerMoved = false;           while (!reader.EOF)         {             if (pointerMoved)             {                 pointerMoved = false;             }             else             {                 if (!reader.Read())                     break;             }               var nodeName = reader.Name;             var nodeType = reader.NodeType;               if (nodeName == "item")             {                 if (nodeType == XmlNodeType.Element)                     item = new FeedItem();                 else if (nodeType == XmlNodeType.EndElement)                     if (item != null)                     {                         items.Add(item);                         item = null;                     }                   continue;             }               if (nodeType != XmlNodeType.Element)                 continue;               if (item == null)                 continue;               reader.MoveToContent();             var nodeValue = reader.ReadElementContentAsString();             // we just moved internal pointer             pointerMoved = true;               if (nodeName == "title")                 item.Title = nodeValue;             else if (nodeName == "description")                 item.Description =  Regex.Replace(nodeValue,@"<(.|\n)*?>",string.Empty);             else if (nodeName == "feedburner:origLink")                 item.Link = nodeValue;             else if (nodeName == "pubDate")             {                 if (!string.IsNullOrEmpty(nodeValue))                     item.PublishDate = DateTime.Parse(nodeValue);             }             else if (nodeName == "category")                 item.Categories.Add(nodeValue);         }           if (ItemsReceived != null)             ItemsReceived(this, items);     } } This method is pretty long but it works. Now let’s try to use it in Windows Phone 7 application. Using RssClient And this is the fragment of code behing the main page of my application start screen. You can see how RssClient is initialized and how items are bound to list that shows them. public MainPage() {     InitializeComponent();       SupportedOrientations = SupportedPageOrientation.Portrait | SupportedPageOrientation.Landscape;     listBox1.Width = Width;       var rssClient = new RssClient("http://feedproxy.google.com/gunnarpeipman");     rssClient.ItemsReceived += new RssClient.ItemsReceivedDelegate(rssClient_ItemsReceived);     rssClient.LoadItems(); }   void rssClient_ItemsReceived(RssClient client, IList<FeedItem> items) {     Dispatcher.BeginInvoke(delegate()     {         listBox1.ItemsSource = items;     });            } Conclusion As you can see it was not very hard task to read RSS-feed and populate list with feed entries. Although we are not able to use more powerful classes that are part of full version on .NET Framework we can still live with limited set of classes that .NET Framework CE provides.

    Read the article

  • Asp.net Google Charts SSL handler for GeoMap

    - by Ian
    Hi All, I am trying to view Google charts in a site using SSL. Google Charts do not support SSL so if we use the standard charts, we get warning messages. My plan is to create a ASHX handler that is co9ntained in the secure site that will retrieve the content from Google and serve this to the page the user is viewing. Using VS 2008 SP1 and the included web server, my idea works perfectly for both Firefox and IE 8 & 9(Preview) and I am able to see my geomap displayed on my page as it should be. But my problem is when I publish to IIS7 the page using my handler to generate the geomap works in Firefox but not IE(every version). There are no errors anywhere or in any log files, but when i right click in IE in the area where the map should be displayed, I see the message in the context menu saying "movie not loaded" Below is the code from my handler and the aspx page. I have disabled compression in my web.config. Even in IE I am hitting all my break points and when I use the IE9 Developer tools, the web page is correctly generated with all the correct code, url's and references. If you have any better ways to accomplish this or how i can fix my problem, I will appreciate it. Thanks Ian Handler(ASHX) public void ProcessRequest(HttpContext context) { String url = "http://charts.apis.google.com/jsapi"; string query = context.Request.QueryString.ToString(); if (!string.IsNullOrEmpty(query)) { url = query; } HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(new Uri(HttpUtility.UrlDecode(url))); request.UserAgent = context.Request.UserAgent; WebResponse response = request.GetResponse(); string PageContent = string.Empty; StreamReader Reader; Stream webStream = response.GetResponseStream(); string contentType = response.ContentType; context.Response.BufferOutput = true; context.Response.ContentType = contentType; context.Response.Cache.SetCacheability(HttpCacheability.NoCache); context.Response.Cache.SetNoServerCaching(); context.Response.Cache.SetMaxAge(System.TimeSpan.Zero); string newUrl = IanLearning.Properties.Settings.Default.HandlerURL; //"https://localhost:444/googlesecurecharts.ashx?"; if (response.ContentType.Contains("javascript")) { Reader = new StreamReader(webStream); PageContent = Reader.ReadToEnd(); PageContent = PageContent.Replace("http://", newUrl + "http://"); PageContent = PageContent.Replace("charts.apis.google.com", newUrl + "charts.apis.google.com"); PageContent = PageContent.Replace(newUrl + "http://maps.google.com/maps/api/", "http://maps.google.com/maps/api/"); context.Response.Write(PageContent); } else { { byte[] bytes = ReadFully(webStream); context.Response.BinaryWrite(bytes); } } context.Response.Flush(); response.Close(); webStream.Close(); context.Response.End(); context.ApplicationInstance.CompleteRequest(); } ASPX Page <%@ Page Title="" Language="C#" MasterPageFile="~/Site2.Master" AutoEventWireup="true" CodeBehind="googlechart.aspx.cs" Inherits="IanLearning.googlechart" %> <asp:Content ID="Content1" ContentPlaceHolderID="head" runat="server"> <script type='text/javascript' src='~/googlesecurecharts.ashx?'></script> <script type='text/javascript'> google.load('visualization', '1', { 'packages': ['geomap'] }); google.setOnLoadCallback(drawMap); var geomap; function drawMap() { var data = new google.visualization.DataTable(); data.addRows(6); data.addColumn('string', 'City'); data.addColumn('number', 'Sales'); data.setValue(0, 0, 'ZA'); data.setValue(0, 1, 200); data.setValue(1, 0, 'US'); data.setValue(1, 1, 300); data.setValue(2, 0, 'BR'); data.setValue(2, 1, 400); data.setValue(3, 0, 'CN'); data.setValue(3, 1, 500); data.setValue(4, 0, 'IN'); data.setValue(4, 1, 600); data.setValue(5, 0, 'ZW'); data.setValue(5, 1, 700); var options = {}; options['region'] = 'world'; options['dataMode'] = 'regions'; options['showZoomOut'] = false; var container = document.getElementById('map_canvas'); geomap = new google.visualization.GeoMap(container); google.visualization.events.addListener( geomap, 'regionClick', function(e) { drillDown(e['region']); }); geomap.draw(data, options); }; function drillDown(regionData) { alert(regionData); } </script> </asp:Content> <asp:Content ID="Content2" ContentPlaceHolderID="ContentPlaceHolder1" runat="server"> <div id='map_canvas'> </div> </asp:Content>

    Read the article

  • The remote server returned an error: (400) Bad Request - uploading less 2MB file size?

    - by fiberOptics
    The file succeed to upload when it is 2KB or lower in size. The main reason why I use streaming is to be able to upload file up to at least 1 GB. But when I try to upload file with less 1MB size, I get bad request. It is my first time to deal with downloading and uploading process, so I can't easily find the cause of error. Testing part: private void button24_Click(object sender, EventArgs e) { try { OpenFileDialog openfile = new OpenFileDialog(); if (openfile.ShowDialog() == System.Windows.Forms.DialogResult.OK) { string port = "3445"; byte[] fileStream; using (FileStream fs = new FileStream(openfile.FileName, FileMode.Open, FileAccess.Read, FileShare.Read)) { fileStream = new byte[fs.Length]; fs.Read(fileStream, 0, (int)fs.Length); fs.Close(); fs.Dispose(); } string baseAddress = "http://localhost:" + port + "/File/AddStream?fileID=9"; HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(baseAddress); request.Method = "POST"; request.ContentType = "text/plain"; //request.ContentType = "application/octet-stream"; Stream serverStream = request.GetRequestStream(); serverStream.Write(fileStream, 0, fileStream.Length); serverStream.Close(); using (HttpWebResponse response = request.GetResponse() as HttpWebResponse) { int statusCode = (int)response.StatusCode; StreamReader reader = new StreamReader(response.GetResponseStream()); } } } catch (Exception ex) { MessageBox.Show(ex.Message); } } Service: [WebInvoke(UriTemplate = "AddStream?fileID={fileID}", Method = "POST", BodyStyle = WebMessageBodyStyle.Bare)] public bool AddStream(long fileID, System.IO.Stream fileStream) { ClasslLogic.FileComponent svc = new ClasslLogic.FileComponent(); return svc.AddStream(fileID, fileStream); } Server code for streaming: namespace ClasslLogic { public class StreamObject : IStreamObject { public bool UploadFile(string filename, Stream fileStream) { try { FileStream fileToupload = new FileStream(filename, FileMode.Create); byte[] bytearray = new byte[10000]; int bytesRead, totalBytesRead = 0; do { bytesRead = fileStream.Read(bytearray, 0, bytearray.Length); totalBytesRead += bytesRead; } while (bytesRead > 0); fileToupload.Write(bytearray, 0, bytearray.Length); fileToupload.Close(); fileToupload.Dispose(); } catch (Exception ex) { throw new Exception(ex.Message); } return true; } } } Web config: <system.serviceModel> <bindings> <basicHttpBinding> <binding> <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="2097152" maxBytesPerRead="4096" maxNameTableCharCount="2097152" /> <security mode="None" /> </binding> <binding name="ClassLogicBasicTransfer" closeTimeout="00:05:00" openTimeout="00:05:00" receiveTimeout="00:15:00" sendTimeout="00:01:00" allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="67108864" maxReceivedMessageSize="67108864" messageEncoding="Mtom" textEncoding="utf-8" useDefaultWebProxy="true"> <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="67108864" maxBytesPerRead="4096" maxNameTableCharCount="67108864" /> <security mode="None"> <transport clientCredentialType="None" proxyCredentialType="None" realm="" /> <message clientCredentialType="UserName" algorithmSuite="Default" /> </security> </binding> <binding name="BaseLogicWSHTTP"> <security mode="None" /> </binding> <binding name="BaseLogicWSHTTPSec" /> </basicHttpBinding> </bindings> <behaviors> <serviceBehaviors> <behavior> <!-- To avoid disclosing metadata information, set the value below to false and remove the metadata endpoint above before deployment --> <serviceMetadata httpGetEnabled="true" /> <!-- To receive exception details in faults for debugging purposes, set the value below to true. Set to false before deployment to avoid disclosing exception information --> <serviceDebug includeExceptionDetailInFaults="true" /> </behavior> </serviceBehaviors> </behaviors> <serviceHostingEnvironment multipleSiteBindingsEnabled="true" aspNetCompatibilityEnabled="true" /> </system.serviceModel> I'm not sure if this affects the streaming function, because I'm using WCF4.0 rest template which config is dependent in Global.asax. One more thing is this, whether I run the service and passing a stream or not, the created file always contain this thing. How could I remove the "NUL" data? Thanks in advance. Edit public bool UploadFile(string filename, Stream fileStream) { try { FileStream fileToupload = new FileStream(filename, FileMode.Create); byte[] bytearray = new byte[10000]; int bytesRead, totalBytesRead = 0; do { bytesRead = fileStream.Read(bytearray, totalBytesRead, bytearray.Length - totalBytesRead); totalBytesRead += bytesRead; } while (bytesRead > 0); fileToupload.Write(bytearray, 0, totalBytesRead); fileToupload.Close(); fileToupload.Dispose(); } catch (Exception ex) { throw new Exception(ex.Message); } return true; }

    Read the article

  • How to add Category in DotClear blog with HttpWebRequest or MetaWeblog API

    - by Pitming
    I'm trying to create/modify dotclear blogs. For most of the options, i use XmlRpc API (DotClear.MetaWeblog). But didn't find any way to handle categories. So I start to look at the Http packet and try to do "the same as the browser". Here si the method I use to "Http POST" protected HttpStatusCode HttpPost(Uri url_, string data_, bool allowAutoRedirect_) { HttpWebRequest Request; HttpWebResponse Response = null; Stream ResponseStream = null; Request = (System.Net.HttpWebRequest)HttpWebRequest.Create(url_); Request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)"; Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"; Request.AllowAutoRedirect = allowAutoRedirect_; // Add the network credentials to the request. Request.Credentials = new NetworkCredential(Username, Password); string authInfo = Username + ":" + Password; authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo)); Request.Headers["Authorization"] = "Basic " + authInfo; Request.Method = "POST"; Request.CookieContainer = Cookies; if(ConnectionCookie!=null) Request.CookieContainer.Add(url_, ConnectionCookie); if (dcAdminCookie != null) Request.CookieContainer.Add(url_, dcAdminCookie); Request.PreAuthenticate = true; ASCIIEncoding encoding = new ASCIIEncoding(); string postData = data_; byte[] data = encoding.GetBytes(postData); //Encoding.UTF8.GetBytes(data_); //encoding.GetBytes(postData); Request.ContentLength = data.Length; Request.ContentType = "application/x-www-form-urlencoded"; Stream newStream = Request.GetRequestStream(); // Send the data. newStream.Write(data, 0, data.Length); newStream.Close(); try { // get the response from the server. Response = (HttpWebResponse)Request.GetResponse(); if (!allowAutoRedirect_) { foreach (Cookie c in Response.Cookies) { if (c.Name == "dcxd") ConnectionCookie = c; if (c.Name == "dc_admin") dcAdminCookie = c; } Cookies.Add(Response.Cookies); } // Get the response stream. ResponseStream = Response.GetResponseStream(); // Pipes the stream to a higher level stream reader with the required encoding format. StreamReader readStream = new StreamReader(ResponseStream, Encoding.UTF8); string result = readStream.ReadToEnd(); if (Request.RequestUri == Response.ResponseUri) { _log.InfoFormat("{0} ==&gt; {1}({2})", Request.RequestUri, Response.StatusCode, Response.StatusDescription); } else { _log.WarnFormat("RequestUri:{0}\r\nResponseUri:{1}\r\nstatus code:{2} Status descr:{3}", Request.RequestUri, Response.ResponseUri, Response.StatusCode, Response.StatusDescription); } } catch (WebException wex) { Response = wex.Response as HttpWebResponse; if (Response != null) { _log.ErrorFormat("{0} ==&gt; {1}({2})", Request.RequestUri, Response.StatusCode, Response.StatusDescription); } Request.Abort(); } finally { if (Response != null) { // Releases the resources of the response. Response.Close(); } } if(Response !=null) return Response.StatusCode; return HttpStatusCode.Ambiguous; } So the first thing to do is to Authenticate as admin. Here is the code: protected bool HttpAuthenticate() { Uri u = new Uri(this.Url); Uri url = new Uri(string.Format("{0}/admin/auth.php", u.GetLeftPart(UriPartial.Authority))); string data = string.Format("user_id={0}&user_pwd={1}&user_remember=1", Username, Password); var ret = HttpPost(url,data,false); return (ret == HttpStatusCode.OK || ret==HttpStatusCode.Found); } 3.Now that I'm authenticate, i need to get a xd_chek info (that i can find on the page so basically it's a GET on /admin/category.php + Regex("dotclear[.]nonce = '(.*)'")) 4.so I'm authenticate and have the xd_check info. The last thing to do seems to post the next category. But of course it does not work at all... here is the code: string postData = string.Format("cat_title={0}&new_cat_parent={1}&xd_check={2}", category_, 0, xdCheck); HttpPost(url, postData, true); If anyone can help me and explain were is it wrong ? thanks in advance.

    Read the article

  • How to create a SOAP REQUEST using ASP.NET (VB) without using Visual

    - by user311691
    Hi all , I urgently need your help . I am new to consuming a web service using SOAP protocol. I have been given a demo webservice URL which ends in .WSDL and NOT .asml?WSDL. The problem is I cannot add a web reference using Visual studio OR Disco.exe or Wsdl.exe - This webservice has been created on a java platform and for security reasons the only way to make a invoke the webservice is at runtime using SOAP protocol IN asp.net (VB). I I have created some code but cannot seem to send the soap object to the receiving web service. If I could get a solution with step by step instructions on how I can send a SOAP REQUEST. Below is my code and all am trying to do is send a SOAP REQUEST and receive a SOAP RESPONSE which I will display in my browser. <%@ page language="vb" %> <%@ Import Namespace="System.Data"%> <%@ Import Namespace="System.Xml"%> <%@ Import Namespace="System.Net"%> <%@ Import Namespace="System.IO"%> <%@ Import Namespace="System.Text"%> <script runat=server> Private Sub Page_Load() Dim objHTTPReq As HttpWebRequest Dim WebserviceUrl As String = "http://xx.xx.xx:8084/asy/wsdl/asy.wsdl" objHTTPReq = CType(WebRequest.Create(WebserviceUrl), HttpWebRequest) Dim soapXML As String soapXML = "<?xml version='1.0' encoding='utf-8'?>" & _ " <soap:Envelope xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'" & _ " xmlns:xsd='http://www.w3.org/2001/XMLSchema'"& _ " xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/' >"& _ " <soap:Body> "& _ " <validatePaymentData xmlns='http://asybanks.webservices.asycuda.org'> " & _ " <bankCode>"& bankCode &"</bankCode> " & _ " <PaymentDataType>" & _ " <paymentType>"& payment_type &"</paymentType> " & _ " <amount>"& ass_amount &"</amount> " & _ " <ReferenceType>" & _ " <year>"& year &"</year> " & _ " <customsOfficeCode>"& station &"</customsOfficeCode> " & _ " </ReferenceType>" & _ " <accountNumber>"& zra_account &"</accountNumber> " & _ " </PaymentDataType> " & _ " </validatePaymentData> " & _ " </soap:Body> " & _ " </soap:Envelope> " objHTTPReq.Headers.Add("SOAPAction", "http://asybanks.webservices.asycuda.org") objHTTPReq.ContentType = "text/xml; charset=utf-8" objHTTPReq.ContentLength = soapXML.Length objHTTPReq.Accept = "text/xml" objHTTPReq.Method = "POST" Dim objHTTPRes As HttpWebResponse = CType(objHTTPReq.GetResponse(), HttpWebResponse) Dim dataStream As Stream = objHTTPRes.GetResponseStream() Dim reader As StreamReader = new StreamReader(dataStream) Dim responseFromServer As String = reader.ReadToEnd() OurXml.text = responseFromServer End Sub </script> <html xmlns="http://www.w3.org/1999/xhtml"> <head runat="server"> <title> XML TRANSACTION SIMULATION - N@W@ TJ </title> </head> <body> <form id="form1" runat="server"> <div> <p>ZRA test Feedback:</p> <asp:label id="OurXml" runat="server"/> </div> </form> </body> </html> the demo webservice looks like this: <?xml version="1.0" encoding="UTF-8" ?> - <!-- WEB SERVICE JAVA DEMO --> - <definitions targetNamespace="http://asybanks.webservices.asycuda.org" xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:apachesoap="http://xml.apache.org/xml-soap" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:y="http://asybanks.webservices.asycuda.org"> - <types> - <xs:schema elementFormDefault="qualified" targetNamespace="http://asybanks.webservices.asycuda.org" xmlns="http://www.w3.org/2001/XMLSchema"> SOME OTHER INFORMATION AT THE BOTTOM <soap:address location="http://xx.xx.xx:8084/asy/services/asy" /> </port> </service> </definitions> From the above excerpt of the wsdl url webservice, I am not sure which namespace to use for soapACTION - please advise.... Please if you could comment every stage of a soap request and provide a working demo - I would be most grateful as I would be learning rather than just assuming stuff :)

    Read the article

  • i need some help with my vb.net codes..plzz

    - by akmalizhar
    currently i need to develop an application that can exctract information from few website.. this is what i have done up until now.. Imports System Imports System.Text.RegularExpressions Imports System.IO Imports System.Net Imports System.Web Imports System.Data.SqlClient Imports System.Threading Imports System.Data.DataSet Imports System.Data.OleDb Module module1 Dim url As String Dim hotelName As String = "" Sub Main() Dim url As String = "" Console.Write("enter url: ") url = Console.ReadLine() extractor(url) End Sub Public Sub extractor(ByVal url As String) Dim strConn As String = "Data Source = localhost; Initial Catalog = knowledgeBase; Integrated Security = True; Connection Timeout = 0;" Dim conn As SqlConnection = New SqlConnection(strConn) conn.Open() Dim strSQL1 As String Dim matchStn1 As String = "" Dim matchstn2 As String = "" Dim matchstn3 As String = "" Dim matchstn4 As String = "" Dim matchstn5 As String = "" Dim matchstn6 As String = "" Dim matchstn7 As String = "" Dim matchstn8 As String = "" Dim matchstn9 As String = "" Dim matchstn10 As String = "" Dim objRequest As WebRequest = HttpWebRequest.Create(url) Dim objResponse As WebResponse = objRequest.GetResponse() Dim objStreamReader As New StreamReader(objResponse.GetResponseStream()) Dim strpage As String = objStreamReader.ReadToEnd Dim RegExStr As String = "<[^>]*>" Dim R As New Regex(RegExStr) Dim sourcestring As String = strpage Dim re As Regex = New Regex("<h2 class=""name hotel""[^>]*>[\s\S]+?</h2>") Dim mc As MatchCollection = re.Matches(sourcestring) Dim mIdx As Integer = 0 For Each m As Match In mc For groupIdx As Integer = 0 To m.Groups.Count - 1 matchStn1 = m.Groups(groupIdx).Value matchStn1 = R.Replace(matchStn1, " ") matchStn1 = matchStn1.Trim() Next mIdx = mIdx + 1 Next Dim re9 As Regex = New Regex("<li class=""cuisine""[^>]*>[^>]+</li>") Dim mc9 As MatchCollection = re9.Matches(sourcestring) Dim mIdx9 As Integer = 0 For Each m As Match In mc9 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn9 = m.Groups(groupIdx).Value matchstn9 = R.Replace(matchstn9, " ") matchstn9 = matchstn9.Trim() Next mIdx = mIdx + 1 Next Dim re2 As Regex = New Regex("<span class=""street-address""[^>]*>[^>]+</span>") Dim mc2 As MatchCollection = re2.Matches(sourcestring) Dim mIdx2 As Integer = 0 For Each m As Match In mc2 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn2 = m.Groups(groupIdx).Value matchstn2 = R.Replace(matchstn2, " ") matchstn2 = matchstn2.Trim() Next mIdx2 = mIdx2 + 1 Next Dim re3 As Regex = New Regex("<span class=""locality""[^>]*>[\s\S]+?</span>") Dim mc3 As MatchCollection = re3.Matches(sourcestring) Dim mIdx3 As Integer = 0 For Each m As Match In mc3 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn3 = m.Groups(groupIdx).Value matchstn3 = R.Replace(matchstn3, " ") matchstn3 = matchstn3.Trim() Next mIdx3 = mIdx3 + 1 Next Dim re4 As Regex = New Regex("<span property=""v:postal-code""[^>]*>[\s\S]+?</span>") Dim mc4 As MatchCollection = re4.Matches(sourcestring) Dim mIdx4 As Integer = 0 For Each m As Match In mc4 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn4 = m.Groups(groupIdx).Value matchstn4 = R.Replace(matchstn4, " ") matchstn4 = matchstn4.Trim() Next mIdx4 = mIdx4 + 1 Next Dim re5 As Regex = New Regex("<span class=""country-name""[^>]*>[\s\S]+?</span>") Dim mc5 As MatchCollection = re5.Matches(sourcestring) Dim mIdx5 As Integer = 0 For Each m As Match In mc5 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn5 = m.Groups(groupIdx).Value matchstn5 = R.Replace(matchstn5, " ") matchstn5 = matchstn5.Trim() Next mIdx5 = mIdx5 + 1 Next Dim re10 As Regex = New Regex("<address class=""adr""[^>]*>[\s\S]+?</address>") Dim mc10 As MatchCollection = re10.Matches(sourcestring) Dim mIdx10 As Integer = 0 For Each m As Match In mc10 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn10 = m.Groups(groupIdx).Value matchstn10 = R.Replace(matchstn10, " ") matchstn10 = matchstn10.Trim() strSQL1 = "insert into infoRestaurant (nameRestaurant, cuisine, streetAddress, locality, postalCode, countryName, addressFull, tel, attractionType) values (N" & _ FormatSqlParam(matchStn1) & ",N" & _ FormatSqlParam(matchstn9) & ",N" & _ FormatSqlParam(matchstn2) & ",N" & _ FormatSqlParam(matchstn3) & ",N" & _ FormatSqlParam(matchstn4) & ",N" & _ FormatSqlParam(matchstn5) & ",N" & _ FormatSqlParam(matchstn10) & ",N" & _ FormatSqlParam(matchstn6) & ",N" & _ FormatSqlParam(matchstn7) & ")" Dim objCommand1 As New SqlCommand(strSQL1, conn) objCommand1.ExecuteNonQuery() Next mIdx4 = mIdx4 + 1 Next Dim re6 As Regex = New Regex("<span class=""tel""[^>]*>[\s\S]+?</span>") Dim mc6 As MatchCollection = re6.Matches(sourcestring) Dim mIdx6 As Integer = 0 For Each m As Match In mc6 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn6 = m.Groups(groupIdx).Value matchstn6 = R.Replace(matchstn6, " ") matchstn6 = matchstn6.Trim() Next mIdx6 = mIdx6 + 1 Next Dim re7 As Regex = New Regex("<div><b>Attraction type:[^>]*>[\s\S]+?</div>") Dim mc7 As MatchCollection = re7.Matches(sourcestring) Dim mIdx7 As Integer = 0 For Each m As Match In mc7 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn7 = m.Groups(groupIdx).Value matchstn7 = R.Replace(matchstn7, " ") matchstn7 = matchstn7.Trim() Next mIdx7 = mIdx7 + 1 Next Dim re8 As Regex = New Regex("(?=<p id).*(?<=</p>)") Dim mc8 As MatchCollection = re8.Matches(sourcestring) Dim mIdx8 As Integer = 0 For Each m As Match In mc8 For groupIdx As Integer = 0 To m.Groups.Count - 1 matchstn8 = m.Groups(groupIdx).Value matchstn8 = R.Replace(matchstn8, " ") matchstn8 = matchstn8.Trim() Dim strSQL2 As String = "insert into feedBackRestaurant (feedBackView) values(N" + FormatSqlParam(matchstn8) + ")" Dim objCommand2 As New SqlCommand(strSQL2, conn) objCommand2.ExecuteNonQuery() Next mIdx8 = mIdx8 + 1 Next objStreamReader.Close() conn.Close() End Sub Public Function FormatSqlParam(ByVal strParam As String) As String Dim newParamFormat As String If strParam = String.Empty Then newParamFormat = "'" & "NA" & "'" Else newParamFormat = strParam.Trim() newParamFormat = "'" & newParamFormat.Replace("'", "''") & "'" End If Return newParamFormat End Function End Module ---problems-- problem that i face are 1. the database foreign key is not working here..someone told me that need some codes to be added..but i dunno how. 2. the data repeats as i run the application. i guest it require update database function.but i hv no idea how. 3. i have to add in multithreading function as well..and last, how to make my application is flexible eventhough the HTML code changes..can anyone help me??plzzz website that i need to extract is http://www.tripadvisor.com/Tourism-g293951-Malaysia-Vacations.html i need the information about hotel, restaurant and attraction place..plzz..i need some help here..

    Read the article

  • Parsing adobe Kuler RSS feed

    - by dezkev
    I have been trying to parse the below XML file (kuler rss feed). I have read the various posts on this site but am unable to piece them together. I specifically want to extract the child(or siblings) nodes of the element <kuler:themeItem>. However I am getting an exception : Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function. Pl help : C# 3.0 net framework 3.5 RSS feed snippet: <?xml version="1.0" encoding="UTF-8" ?> - <rss xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:kuler="http://kuler.adobe.com/kuler/API/rss/" xmlns:rss="http://blogs.law.harvard.edu/tech/rss" version="2.0"> - <channel> <title>kuler recent themes</title> <link>http://kuler.adobe.com/</link> <description>most recent themes published on kuler (1 to 20 of 332518)</description> <language>en-us</language> <pubDate>Wed, 07 Apr 2010 08:41:31 PST</pubDate> <lastBuildDate>Wed, 07 Apr 2010 08:41:31 PST</lastBuildDate> <docs>http://blogs.law.harvard.edu/tech/rss</docs> <generator>Kuler Services</generator> <managingEditor>[email protected]</managingEditor> <webMaster>[email protected]</webMaster> <recordCount>332518</recordCount> <startIndex>0</startIndex> <itemsPerPage>20</itemsPerPage> - <item> <title>Theme Title: Muted Graph</title> <link>http://kuler.adobe.com/index.cfm#themeID/856075</link> <guid>http://kuler.adobe.com/index.cfm#themeID/856075</guid> - <enclosure xmlns="http://www.solitude.dk/syndication/enclosures/"> <title>Muted Graph</title> - <link length="1" type="image/png"> <url>http://kuler-api.adobe.com/kuler/themeImages/theme_856075.png</url> </link> </enclosure> <description><img src="http://kuler-api.adobe.com/kuler/themeImages/theme_856075.png" /><br /> Artist: tischt<br /> ThemeID: 856075<br /> Posted: 04/07/2010<br /> Hex: F1E9B2, 3D3606, 2A3231, 4A0A07, 424431</description> - <kuler:themeItem> <kuler:themeID>856075</kuler:themeID> <kuler:themeTitle>Muted Graph</kuler:themeTitle> <kuler:themeImage>http://kuler-api.adobe.com/kuler/themeImages/theme_856075.png</kuler:themeImage> - <kuler:themeAuthor> <kuler:authorID>216099</kuler:authorID> <kuler:authorLabel>tischt</kuler:authorLabel> </kuler:themeAuthor> <kuler:themeTags /> <kuler:themeRating>0</kuler:themeRating> <kuler:themeDownloadCount>0</kuler:themeDownloadCount> <kuler:themeCreatedAt>20100407</kuler:themeCreatedAt> <kuler:themeEditedAt>20100407</kuler:themeEditedAt> - <kuler:themeSwatches> - <kuler:swatch> <kuler:swatchHexColor>F1E9B2</kuler:swatchHexColor> <kuler:swatchColorMode>rgb</kuler:swatchColorMode> <kuler:swatchChannel1>0.945098</kuler:swatchChannel1> <kuler:swatchChannel2>0.913725</kuler:swatchChannel2> <kuler:swatchChannel3>0.698039</kuler:swatchChannel3> <kuler:swatchChannel4>0.0</kuler:swatchChannel4> <kuler:swatchIndex>0</kuler:swatchIndex> </kuler:swatch> My Code so far: static void Main(string[] args) { const string feedUrl = "http://kuler-api.adobe.com/rss/get.cfm?listtype=recent&key=xxxx"; var doc = new XmlDocument(); var request = WebRequest.Create(feedUrl) as HttpWebRequest; using (var response = request.GetResponse() as HttpWebResponse) { var reader = new StreamReader(response.GetResponseStream()); doc.Load(reader); } XmlNodeList rsslist = doc.SelectNodes("//rss/channel/item/kuler:themeItem"); for (int i = 0; i < rsslist.Count; i++) { XmlNode rssdetail = rsslist.Item(i).SelectSingleNode("kuler:themeTitle"); string title = rssdetail.InnerText; Console.WriteLine(title); } } }

    Read the article

  • Asynchronous Webcrawling F#, something wrong ?

    - by jlezard
    Not quite sure if it is ok to do this but, my question is: Is there something wrong with my code ? It doesn't go as fast as I would like, and since I am using lots of async workflows maybe I am doing something wrong. The goal here is to build something that can crawl 20 000 pages in less than an hour. open System open System.Text open System.Net open System.IO open System.Text.RegularExpressions open System.Collections.Generic open System.ComponentModel open Microsoft.FSharp open System.Threading //This is the Parallel.Fs file type ComparableUri ( uri: string ) = inherit System.Uri( uri ) let elts (uri:System.Uri) = uri.Scheme, uri.Host, uri.Port, uri.Segments interface System.IComparable with member this.CompareTo( uri2 ) = compare (elts this) (elts(uri2 :?> ComparableUri)) override this.Equals(uri2) = compare this (uri2 :?> ComparableUri ) = 0 override this.GetHashCode() = 0 ///////////////////////////////////////////////Funtions to retreive html string////////////////////////////// let mutable error = Set.empty<ComparableUri> let mutable visited = Set.empty<ComparableUri> let getHtmlPrimitiveAsyncDelay (delay:int) (uri : ComparableUri) = async{ try let req = (WebRequest.Create(uri)) :?> HttpWebRequest // 'use' is equivalent to ‘using’ in C# for an IDisposable req.UserAgent<-"Mozilla" //Console.WriteLine("Waiting") do! Async.Sleep(delay * 250) let! resp = (req.AsyncGetResponse()) Console.WriteLine(uri.AbsoluteUri+" got response after delay "+string delay) use stream = resp.GetResponseStream() use reader = new StreamReader(stream) let html = reader.ReadToEnd() return html with | _ as ex -> Console.WriteLine( ex.ToString() ) lock error (fun () -> error<- error.Add uri ) lock visited (fun () -> visited<-visited.Add uri ) return "BadUri" } ///////////////////////////////////////////////Active Pattern Matching to retreive href////////////////////////////// let (|Matches|_|) (pat:string) (inp:string) = let m = Regex.Matches(inp, pat) // Note the List.tl, since the first group is always the entirety of the matched string. if m.Count > 0 then Some (List.tail [ for g in m -> g.Value ]) else None let (|Match|_|) (pat:string) (inp:string) = let m = Regex.Match(inp, pat) // Note the List.tl, since the first group is always the entirety of the matched string. if m.Success then Some (List.tail [ for g in m.Groups -> g.Value ]) else None ///////////////////////////////////////////////Find Bad href////////////////////////////// let isEmail (link:string) = link.Contains("@") let isMailto (link:string) = if Seq.length link >=6 then link.[0..5] = "mailto" else false let isJavascript (link:string) = if Seq.length link >=10 then link.[0..9] = "javascript" else false let isBadUri (link:string) = link="BadUri" let isEmptyHttp (link:string) = link="http://" let isFile (link:string)= if Seq.length link >=6 then link.[0..5] = "file:/" else false let containsPipe (link:string) = link.Contains("|") let isAdLink (link:string) = if Seq.length link >=6 then link.[0..5] = "adlink" elif Seq.length link >=9 then link.[0..8] = "http://adLink" else false ///////////////////////////////////////////////Find Bad href////////////////////////////// let getHref (htmlString:string) = let urlPat = "href=\"([^\"]+)" match htmlString with | Matches urlPat urls -> urls |> List.map( fun href -> match href with | Match (urlPat) (link::[]) -> link | _ -> failwith "The href was not in correct format, there was more than one match" ) | _ -> Console.WriteLine( "No links for this page" );[] |> List.filter( fun link -> not(isEmail link) ) |> List.filter( fun link -> not(isMailto link) ) |> List.filter( fun link -> not(isJavascript link) ) |> List.filter( fun link -> not(isBadUri link) ) |> List.filter( fun link -> not(isEmptyHttp link) ) |> List.filter( fun link -> not(isFile link) ) |> List.filter( fun link -> not(containsPipe link) ) |> List.filter( fun link -> not(isAdLink link) ) let treatAjax (href:System.Uri) = let link = href.ToString() let firstPart = (link.Split([|"#"|],System.StringSplitOptions.None)).[0] new Uri(firstPart) //only follow pages with certain extnsion or ones with no exensions let followHref (href:System.Uri) = let valid2 = set[".py"] let valid3 = set[".php";".htm";".asp"] let valid4 = set[".php3";".php4";".php5";".html";".aspx"] let arrLength = href.Segments |> Array.length let lastExtension = (href.Segments).[arrLength-1] let lengthLastExtension = Seq.length lastExtension if (lengthLastExtension <= 3) then not( lastExtension.Contains(".") ) else //test for the 2 case let last4 = lastExtension.[(lengthLastExtension-1)-3..(lengthLastExtension-1)] let isValid2 = valid2|>Seq.exists(fun validEnd -> last4.EndsWith( validEnd) ) if isValid2 then true else if lengthLastExtension <= 4 then not( last4.Contains(".") ) else let last5 = lastExtension.[(lengthLastExtension-1)-4..(lengthLastExtension-1)] let isValid3 = valid3|>Seq.exists(fun validEnd -> last5.EndsWith( validEnd) ) if isValid3 then true else if lengthLastExtension <= 5 then not( last5.Contains(".") ) else let last6 = lastExtension.[(lengthLastExtension-1)-5..(lengthLastExtension-1)] let isValid4 = valid4|>Seq.exists(fun validEnd -> last6.EndsWith( validEnd) ) if isValid4 then true else not( last6.Contains(".") ) && not(lastExtension.[0..5] = "mailto") //Create the correct links / -> add the homepage , make them a comparabel Uri let hrefLinksToUri ( uri:ComparableUri ) (hrefLinks:string list) = hrefLinks |> List.map( fun link -> try if Seq.length link <4 then Some(new Uri( uri, link )) else if link.[0..3] = "http" then Some(new Uri(link)) else Some(new Uri( uri, link )) with | _ as ex -> Console.WriteLine(link); lock error (fun () ->error<-error.Add uri) None ) |> List.filter( fun link -> link.IsSome ) |> List.map( fun o -> o.Value) |> List.map( fun uri -> new ComparableUri( string uri ) ) //Treat uri , removing ajax last part , and only following links specified b Benoit let linksToFollow (hrefUris:ComparableUri list) = hrefUris |>List.map( treatAjax ) |>List.filter( fun link -> followHref link ) |>List.map( fun uri -> new ComparableUri( string uri ) ) |>Set.ofList let needToVisit uri = ( lock visited (fun () -> not( visited.Contains uri) ) ) && (lock error (fun () -> not( error.Contains uri) )) let getLinksToFollowAsyncDelay (delay:int) ( uri: ComparableUri ) = async{ let! links = getHtmlPrimitiveAsyncDelay delay uri lock visited (fun () ->visited<-visited.Add uri) let linksToFollow = getHref links |> hrefLinksToUri uri |> linksToFollow |> Set.filter( needToVisit ) |> Set.map( fun link -> if uri.Authority=link.Authority then link else link ) return linksToFollow } //Add delays if visitng same authority let getDelay(uri:ComparableUri) (authorityDelay:Dictionary<string,int>) = let uriAuthority = uri.Authority let hasAuthority,delay = authorityDelay.TryGetValue(uriAuthority) if hasAuthority then authorityDelay.[uriAuthority] <-delay+1 delay else authorityDelay.Add(uriAuthority,1) 0 let rec getLinksToFollowFromSetAsync maxIteration ( uris: seq<ComparableUri> ) = let authorityDelay = Dictionary<string,int>() if maxIteration = 100 then Console.WriteLine("Finished") else //Unite by authority add delay for those we same authority others ignore let stopwatch= System.Diagnostics.Stopwatch() stopwatch.Start() let newLinks = uris |> Seq.map( fun uri -> let delay = lock authorityDelay (fun () -> getDelay uri authorityDelay ) getLinksToFollowAsyncDelay delay uri ) |> Async.Parallel |> Async.RunSynchronously |> Seq.concat stopwatch.Stop() Console.WriteLine("\n\n\n\n\n\n\nTimeElapse : "+string stopwatch.Elapsed+"\n\n\n\n\n\n\n\n\n") getLinksToFollowFromSetAsync (maxIteration+1) newLinks getLinksToFollowFromSetAsync 0 (seq[ComparableUri( "http://twitter.com/" )]) Console.WriteLine("Finished") Some feedBack would be great ! Thank you (note this is just something I am doing for fun)

    Read the article

  • Parse XML document

    - by Neil
    I am trying to parse a remote XML document (from Amazon AWS): <ItemLookupResponse xmlns="http://webservices.amazon.com/AWSECommerceService/2009-03-31"> <OperationRequest> <RequestId>011d32c5-4fab-4c7d-8785-ac48b9bda6da</RequestId> <Arguments> <Argument Name="Condition" Value="New"></Argument> <Argument Name="Operation" Value="ItemLookup"></Argument> <Argument Name="Service" Value="AWSECommerceService"></Argument> <Argument Name="Signature" Value="73l8oLJhITTsWtHxsdrS3BMKsdf01n37PE8u/XCbsJM="></Argument> <Argument Name="MerchantId" Value="Amazon"></Argument> <Argument Name="Version" Value="2009-03-31"></Argument> <Argument Name="ItemId" Value="603084260089"></Argument> <Argument Name="IdType" Value="UPC"></Argument> <Argument Name="AWSAccessKeyId" Value="[myAccessKey]"></Argument> <Argument Name="Timestamp" Value="2010-06-14T15:03:27Z"></Argument> <Argument Name="ResponseGroup" Value="OfferSummary,ItemAttributes"></Argument> <Argument Name="SearchIndex" Value="All"></Argument> </Arguments> <RequestProcessingTime>0.0318510000000000</RequestProcessingTime> </OperationRequest> <Items> <Request> <IsValid>True</IsValid> <ItemLookupRequest> <Condition>New</Condition> <DeliveryMethod>Ship</DeliveryMethod> <IdType>UPC</IdType> <MerchantId>Amazon</MerchantId> <OfferPage>1</OfferPage> <ItemId>603084260089</ItemId> <ResponseGroup>OfferSummary</ResponseGroup> <ResponseGroup>ItemAttributes</ResponseGroup> <ReviewPage>1</ReviewPage> <ReviewSort>-SubmissionDate</ReviewSort> <SearchIndex>All</SearchIndex> <VariationPage>All</VariationPage> </ItemLookupRequest> </Request> <Item> <ASIN>B0000UTUNI</ASIN> <DetailPageURL>http://www.amazon.com/Garnier-Fructis-Fortifying-Conditioner-Minute/dp/B0000UTUNI%3FSubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26creativeASIN%3DB0000UTUNI</DetailPageURL> <ItemLinks> <ItemLink> <Description>Technical Details</Description> <URL>http://www.amazon.com/Garnier-Fructis-Fortifying-Conditioner-Minute/dp/tech-data/B0000UTUNI%3FSubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> <ItemLink> <Description>Add To Baby Registry</Description> <URL>http://www.amazon.com/gp/registry/baby/add-item.html%3Fasin.0%3DB0000UTUNI%26SubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> <ItemLink> <Description>Add To Wedding Registry</Description> <URL>http://www.amazon.com/gp/registry/wedding/add-item.html%3Fasin.0%3DB0000UTUNI%26SubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> <ItemLink> <Description>Add To Wishlist</Description> <URL>http://www.amazon.com/gp/registry/wishlist/add-item.html%3Fasin.0%3DB0000UTUNI%26SubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> <ItemLink> <Description>Tell A Friend</Description> <URL>http://www.amazon.com/gp/pdp/taf/B0000UTUNI%3FSubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> <ItemLink> <Description>All Customer Reviews</Description> <URL>http://www.amazon.com/review/product/B0000UTUNI%3FSubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> <ItemLink> <Description>All Offers</Description> <URL>http://www.amazon.com/gp/offer-listing/B0000UTUNI%3FSubscriptionId%3DAKIAIYPTKHCWTRWWPWBQ%26tag%3Dws%26linkCode%3Dxm2%26camp%3D2025%26creative%3D386001%26creativeASIN%3DB0000UTUNI</URL> </ItemLink> </ItemLinks> <ItemAttributes> <Binding>Health and Beauty</Binding> <Brand>Garnier</Brand> <EAN>0603084260089</EAN> <Feature>Helps restore strength and shine</Feature> <Feature>Penetrates deep to nourish, repair and rejuvenate</Feature> <Feature>Makes hair softer and more manageable without weighing it down</Feature> <ItemDimensions> <Weight Units="hundredths-pounds">40</Weight> </ItemDimensions> <Label>Garnier</Label> <ListPrice> <Amount>419</Amount> <CurrencyCode>USD</CurrencyCode> <FormattedPrice>$4.19</FormattedPrice> </ListPrice> <Manufacturer>Garnier</Manufacturer> <NumberOfItems>1</NumberOfItems> <ProductGroup>Health and Beauty</ProductGroup> <ProductTypeName>ABIS_DRUGSTORE</ProductTypeName> <Publisher>Garnier</Publisher> <Size>5.0 oz</Size> <Studio>Garnier</Studio> <Title>Garnier Fructis Fortifying Fortifying Deep Conditioner, 3 Minute Masque - 5 oz</Title> <UPC>603084260089</UPC> </ItemAttributes> <OfferSummary> <LowestNewPrice> <Amount>229</Amount> <CurrencyCode>USD</CurrencyCode> <FormattedPrice>$2.29</FormattedPrice> </LowestNewPrice> <TotalNew>7</TotalNew> <TotalUsed>0</TotalUsed> <TotalCollectible>0</TotalCollectible> <TotalRefurbished>0</TotalRefurbished> </OfferSummary> </Item> </Items> </ItemLookupResponse> I am trying to extract data from the XML stream using XPathDocument, but with no luck: WebRequest request = HttpWebRequest.Create(url); WebResponse response = request.GetResponse(); //XmlDocument doc = new XmlDocument(); XPathDocument Doc = new XPathDocument(response.GetResponseStream()); XPathNavigator nav = Doc.CreateNavigator(); XPathNodeIterator ListPrice = nav.Select("/ItemLookupResponse/Items/Item/ItemAttributes/ListPrice"); foreach (XPathNavigator node in ListPrice) { Response.Write(node.GetAttribute("Amount", NAMESPACE)); } What am I missing? Thanks in advance!!

    Read the article

  • Building a better mouse-trap &ndash; Improving the creation of XML Message Requests using Reflection, XML &amp; XSLT

    - by paulschapman
    Introduction The way I previously created messages to send to the GovTalk service I used the XMLDocument to create the request. While this worked it left a number of problems; not least that for every message a special function would need to created. This is OK for the short term but the biggest cost in any software project is maintenance and this would be a headache to maintain. So the following is a somewhat better way of achieving the same thing. For the purposes of this article I am going to be using the CompanyNumberSearch request of the GovTalk service – although this technique would work for any service that accepted XML. The C# functions which send and receive the messages remain the same. The magic sauce in this is the XSLT which defines the structure of the request, and the use of objects in conjunction with reflection to provide the content. It is a bit like Sweet Chilli Sauce added to Chicken on a bed of rice. So on to the Sweet Chilli Sauce The Sweet Chilli Sauce The request to search for a company based on it’s number is as follows; <GovTalkMessage xsi:schemaLocation="http://www.govtalk.gov.uk/CM/envelope http://xmlgw.companieshouse.gov.uk/v1-0/schema/Egov_ch-v2-0.xsd" xmlns="http://www.govtalk.gov.uk/CM/envelope" xmlns:dsig="http://www.w3.org/2000/09/xmldsig#" xmlns:gt="http://www.govtalk.gov.uk/schemas/govtalk/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > <EnvelopeVersion>1.0</EnvelopeVersion> <Header> <MessageDetails> <Class>NumberSearch</Class> <Qualifier>request</Qualifier> <TransactionID>1</TransactionID> </MessageDetails> <SenderDetails> <IDAuthentication> <SenderID>????????????????????????????????</SenderID> <Authentication> <Method>CHMD5</Method> <Value>????????????????????????????????</Value> </Authentication> </IDAuthentication> </SenderDetails> </Header> <GovTalkDetails> <Keys/> </GovTalkDetails> <Body> <NumberSearchRequest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xmlgw.companieshouse.gov.uk/v1-0/schema/NumberSearch.xsd"> <PartialCompanyNumber>99999999</PartialCompanyNumber> <DataSet>LIVE</DataSet> <SearchRows>1</SearchRows> </NumberSearchRequest> </Body> </GovTalkMessage> This is the XML that we send to the GovTalk Service and we get back a list of companies that match the criteria passed A message is structured in two parts; The envelope which identifies the person sending the request, with the name of the request, and the body which gives the detail of the company we are looking for. The Chilli What makes it possible is the use of XSLT to define the message – and serialization to convert each request object into XML. To start we need to create an object which will represent the contents of the message we are sending. However there is a common properties in all the messages that we send to Companies House. These properties are as follows SenderId – the id of the person sending the message SenderPassword – the password associated with Id TransactionId – Unique identifier for the message AuthenticationValue – authenticates the request Because these properties are unique to the Companies House message, and because they are shared with all messages they are perfect candidates for a base class. The class is as follows; using System; using System.Collections.Generic; using System.Linq; using System.Web; using System.Security.Cryptography; using System.Text; using System.Text.RegularExpressions; using Microsoft.WindowsAzure.ServiceRuntime; namespace CompanyHub.Services { public class GovTalkRequest { public GovTalkRequest() { try { SenderID = RoleEnvironment.GetConfigurationSettingValue("SenderId"); SenderPassword = RoleEnvironment.GetConfigurationSettingValue("SenderPassword"); TransactionId = DateTime.Now.Ticks.ToString(); AuthenticationValue = EncodePassword(String.Format("{0}{1}{2}", SenderID, SenderPassword, TransactionId)); } catch (System.Exception ex) { throw ex; } } /// <summary> /// returns the Sender ID to be used when communicating with the GovTalk Service /// </summary> public String SenderID { get; set; } /// <summary> /// return the password to be used when communicating with the GovTalk Service /// </summary> public String SenderPassword { get; set; } // end SenderPassword /// <summary> /// Transaction Id - uses the Time and Date converted to Ticks /// </summary> public String TransactionId { get; set; } // end TransactionId /// <summary> /// calculate the authentication value that will be used when /// communicating with /// </summary> public String AuthenticationValue { get; set; } // end AuthenticationValue property /// <summary> /// encodes password(s) using MD5 /// </summary> /// <param name="clearPassword"></param> /// <returns></returns> public static String EncodePassword(String clearPassword) { MD5CryptoServiceProvider md5Hasher = new MD5CryptoServiceProvider(); byte[] hashedBytes; UTF32Encoding encoder = new UTF32Encoding(); hashedBytes = md5Hasher.ComputeHash(ASCIIEncoding.Default.GetBytes(clearPassword)); String result = Regex.Replace(BitConverter.ToString(hashedBytes), "-", "").ToLower(); return result; } } } There is nothing particularly clever here, except for the EncodePassword method which hashes the value made up of the SenderId, Password and Transaction id. Each message inherits from this object. So for the Company Number Search in addition to the properties above we need a partial number, which dataset to search – for the purposes of the project we only need to search the LIVE set so this can be set in the constructor and the SearchRows. Again all are set as properties. With the SearchRows and DataSet initialized in the constructor. public class CompanyNumberSearchRequest : GovTalkRequest, IDisposable { /// <summary> /// /// </summary> public CompanyNumberSearchRequest() : base() { DataSet = "LIVE"; SearchRows = 1; } /// <summary> /// Company Number to search against /// </summary> public String PartialCompanyNumber { get; set; } /// <summary> /// What DataSet should be searched for the company /// </summary> public String DataSet { get; set; } /// <summary> /// How many rows should be returned /// </summary> public int SearchRows { get; set; } public void Dispose() { DataSet = String.Empty; PartialCompanyNumber = String.Empty; DataSet = "LIVE"; SearchRows = 1; } } As well as inheriting from our base class, I have also inherited from IDisposable – not just because it is just plain good practice to dispose of objects when coding, but it gives also gives us more versatility when using the object. There are four stages in making a request and this is reflected in the four methods we execute in making a call to the Companies House service; Create a request Send a request Check the status If OK then get the results of the request I’ve implemented each of these stages within a static class called Toolbox – which also means I don’t need to create an instance of the class to use it. When making a request there are three stages; Get the template for the message Serialize the object representing the message Transform the serialized object using a predefined XSLT file. Each of my templates I have defined as an embedded resource. When retrieving a resource of this kind we have to include the full namespace to the resource. In making the code re-usable as much as possible I defined the full ‘path’ within the GetRequest method. requestFile = String.Format("CompanyHub.Services.Schemas.{0}", RequestFile); So we now have the full path of the file within the assembly. Now all we need do is retrieve the assembly and get the resource. asm = Assembly.GetExecutingAssembly(); sr = asm.GetManifestResourceStream(requestFile); Once retrieved  So this can be returned to the calling function and we now have a stream of XSLT to define the message. Time now to serialize the request to create the other side of this message. // Serialize object containing Request, Load into XML Document t = Obj.GetType(); ms = new MemoryStream(); serializer = new XmlSerializer(t); xmlTextWriter = new XmlTextWriter(ms, Encoding.ASCII); serializer.Serialize(xmlTextWriter, Obj); ms = (MemoryStream)xmlTextWriter.BaseStream; GovTalkRequest = Toolbox.ConvertByteArrayToString(ms.ToArray()); First off we need the type of the object so we make a call to the GetType method of the object containing the Message properties. Next we need a MemoryStream, XmlSerializer and an XMLTextWriter so these can be initialized. The object is serialized by making the call to the Serialize method of the serializer object. The result of that is then converted into a MemoryStream. That MemoryStream is then converted into a string. ConvertByteArrayToString This is a fairly simple function which uses an ASCIIEncoding object found within the System.Text namespace to convert an array of bytes into a string. public static String ConvertByteArrayToString(byte[] bytes) { System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding(); return enc.GetString(bytes); } I only put it into a function because I will be using this in various places. The Sauce When adding support for other messages outside of creating a new object to store the properties of the message, the C# components do not need to change. It is in the XSLT file that the versatility of the technique lies. The XSLT file determines the format of the message. For the CompanyNumberSearch the XSLT file is as follows; <?xml version="1.0"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="/"> <GovTalkMessage xsi:schemaLocation="http://www.govtalk.gov.uk/CM/envelope http://xmlgw.companieshouse.gov.uk/v1-0/schema/Egov_ch-v2-0.xsd" xmlns="http://www.govtalk.gov.uk/CM/envelope" xmlns:dsig="http://www.w3.org/2000/09/xmldsig#" xmlns:gt="http://www.govtalk.gov.uk/schemas/govtalk/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > <EnvelopeVersion>1.0</EnvelopeVersion> <Header> <MessageDetails> <Class>NumberSearch</Class> <Qualifier>request</Qualifier> <TransactionID> <xsl:value-of select="CompanyNumberSearchRequest/TransactionId"/> </TransactionID> </MessageDetails> <SenderDetails> <IDAuthentication> <SenderID><xsl:value-of select="CompanyNumberSearchRequest/SenderID"/></SenderID> <Authentication> <Method>CHMD5</Method> <Value> <xsl:value-of select="CompanyNumberSearchRequest/AuthenticationValue"/> </Value> </Authentication> </IDAuthentication> </SenderDetails> </Header> <GovTalkDetails> <Keys/> </GovTalkDetails> <Body> <NumberSearchRequest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xmlgw.companieshouse.gov.uk/v1-0/schema/NumberSearch.xsd"> <PartialCompanyNumber> <xsl:value-of select="CompanyNumberSearchRequest/PartialCompanyNumber"/> </PartialCompanyNumber> <DataSet> <xsl:value-of select="CompanyNumberSearchRequest/DataSet"/> </DataSet> <SearchRows> <xsl:value-of select="CompanyNumberSearchRequest/SearchRows"/> </SearchRows> </NumberSearchRequest> </Body> </GovTalkMessage> </xsl:template> </xsl:stylesheet> The outer two tags define that this is a XSLT stylesheet and the root tag from which the nodes are searched for. The GovTalkMessage is the format of the message that will be sent to Companies House. We first set up the XslCompiledTransform object which will transform the XSLT template and the serialized object into the request to Companies House. xslt = new XslCompiledTransform(); resultStream = new MemoryStream(); writer = new XmlTextWriter(resultStream, Encoding.ASCII); doc = new XmlDocument(); The Serialize method require XmlTextWriter to write the XML (writer) and a stream to place the transferred object into (writer). The XML will be loaded into an XMLDocument object (doc) prior to the transformation. // create XSLT Template xslTemplate = Toolbox.GetRequest(Template); xslTemplate.Seek(0, SeekOrigin.Begin); templateReader = XmlReader.Create(xslTemplate); xslt.Load(templateReader); I have stored all the templates as a series of Embedded Resources and the GetRequestCall takes the name of the template and extracts the relevent XSLT file. /// <summary> /// Gets the framwork XML which makes the request /// </summary> /// <param name="RequestFile"></param> /// <returns></returns> public static Stream GetRequest(String RequestFile) { String requestFile = String.Empty; Stream sr = null; Assembly asm = null; try { requestFile = String.Format("CompanyHub.Services.Schemas.{0}", RequestFile); asm = Assembly.GetExecutingAssembly(); sr = asm.GetManifestResourceStream(requestFile); } catch (Exception) { throw; } finally { asm = null; } return sr; } // end private static stream GetRequest We first take the template name and expand it to include the full namespace to the Embedded Resource I like to keep all my schemas in the same directory and so the namespace reflects this. The rest is the default namespace for the project. Then we get the currently executing assembly (which will contain the resources with the call to GetExecutingAssembly() ) Finally we get a stream which contains the XSLT file. We use this stream and then load an XmlReader with the contents of the template, and that is in turn loaded into the XslCompiledTransform object. We convert the object containing the message properties into Xml by serializing it; calling the Serialize() method of the XmlSerializer object. To set up the object we do the following; t = Obj.GetType(); ms = new MemoryStream(); serializer = new XmlSerializer(t); xmlTextWriter = new XmlTextWriter(ms, Encoding.ASCII); We first determine the type of the object being transferred by calling GetType() We create an XmlSerializer object by passing the type of the object being serialized. The serializer writes to a memory stream and that is linked to an XmlTextWriter. Next job is to serialize the object and load it into an XmlDocument. serializer.Serialize(xmlTextWriter, Obj); ms = (MemoryStream)xmlTextWriter.BaseStream; xmlRequest = new XmlTextReader(ms); GovTalkRequest = Toolbox.ConvertByteArrayToString(ms.ToArray()); doc.LoadXml(GovTalkRequest); Time to transform the XML to construct the full request. xslt.Transform(doc, writer); resultStream.Seek(0, SeekOrigin.Begin); request = Toolbox.ConvertByteArrayToString(resultStream.ToArray()); So that creates the full request to be sent  to Companies House. Sending the request So far we have a string with a request for the Companies House service. Now we need to send the request to the Companies House Service. Configuration within an Azure project There are entire blog entries written about configuration within an Azure project – most of this is out of scope for this article but the following is a summary. Configuration is defined in two files within the parent project *.csdef which contains the definition of configuration setting. <?xml version="1.0" encoding="utf-8"?> <ServiceDefinition name="OnlineCompanyHub" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition"> <WebRole name="CompanyHub.Host"> <InputEndpoints> <InputEndpoint name="HttpIn" protocol="http" port="80" /> </InputEndpoints> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" /> <Setting name="DataConnectionString" /> </ConfigurationSettings> </WebRole> <WebRole name="CompanyHub.Services"> <InputEndpoints> <InputEndpoint name="HttpIn" protocol="http" port="8080" /> </InputEndpoints> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" /> <Setting name="SenderId"/> <Setting name="SenderPassword" /> <Setting name="GovTalkUrl"/> </ConfigurationSettings> </WebRole> <WorkerRole name="CompanyHub.Worker"> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" /> </ConfigurationSettings> </WorkerRole> </ServiceDefinition>   Above is the configuration definition from the project. What we are interested in however is the ConfigurationSettings tag of the CompanyHub.Services WebRole. There are four configuration settings here, but at the moment we are interested in the second to forth settings; SenderId, SenderPassword and GovTalkUrl The value of these settings are defined in the ServiceDefinition.cscfg file; <?xml version="1.0"?> <ServiceConfiguration serviceName="OnlineCompanyHub" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration"> <Role name="CompanyHub.Host"> <Instances count="2" /> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" /> <Setting name="DataConnectionString" value="UseDevelopmentStorage=true" /> </ConfigurationSettings> </Role> <Role name="CompanyHub.Services"> <Instances count="2" /> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" /> <Setting name="SenderId" value="UserID"/> <Setting name="SenderPassword" value="Password"/> <Setting name="GovTalkUrl" value="http://xmlgw.companieshouse.gov.uk/v1-0/xmlgw/Gateway"/> </ConfigurationSettings> </Role> <Role name="CompanyHub.Worker"> <Instances count="2" /> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" /> </ConfigurationSettings> </Role> </ServiceConfiguration>   Look for the Role tag that contains our project name (CompanyHub.Services). Having configured the parameters we can now transmit the request. This is done by ‘POST’ing a stream of XML to the Companies House servers. govTalkUrl = RoleEnvironment.GetConfigurationSettingValue("GovTalkUrl"); request = WebRequest.Create(govTalkUrl); request.Method = "POST"; request.ContentType = "text/xml"; writer = new StreamWriter(request.GetRequestStream()); writer.WriteLine(RequestMessage); writer.Close(); We use the WebRequest object to send the object. Set the method of sending to ‘POST’ and the type of data as text/xml. Once set up all we do is write the request to the writer – this sends the request to Companies House. Did the Request Work Part I – Getting the response Having sent a request – we now need the result of that request. response = request.GetResponse(); reader = response.GetResponseStream(); result = Toolbox.ConvertByteArrayToString(Toolbox.ReadFully(reader));   The WebRequest object has a GetResponse() method which allows us to get the response sent back. Like many of these calls the results come in the form of a stream which we convert into a string. Did the Request Work Part II – Translating the Response Much like XSLT and XML were used to create the original request, so it can be used to extract the response and by deserializing the result we create an object that contains the response. Did it work? It would be really great if everything worked all the time. Of course if it did then I don’t suppose people would pay me and others the big bucks so that our programmes do not a) Collapse in a heap (this is an area of memory) b) Blow every fuse in the place in a shower of sparks (this will probably not happen this being real life and not a Hollywood movie, but it was possible to blow the sound system of a BBC Model B with a poorly coded setting) c) Go nuts and trap everyone outside the airlock (this was from a movie, and unless NASA get a manned moon/mars mission set up unlikely to happen) d) Go nuts and take over the world (this was also from a movie, but please note life has a habit of being of exceeding the wildest imaginations of Hollywood writers (note writers – Hollywood executives have no imagination and judging by recent output of that town have turned plagiarism into an art form). e) Freeze in total confusion because the cleaner pulled the plug to the internet router (this has happened) So anyway – we need to check to see if our request actually worked. Within the GovTalk response there is a section that details the status of the message and a description of what went wrong (if anything did). I have defined an XSLT template which will extract these into an XML document. <?xml version="1.0"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:ev="http://www.govtalk.gov.uk/CM/envelope" xmlns:gt="http://www.govtalk.gov.uk/schemas/govtalk/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <xsl:template match="/"> <GovTalkStatus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <Status> <xsl:value-of select="ev:GovTalkMessage/ev:Header/ev:MessageDetails/ev:Qualifier"/> </Status> <Text> <xsl:value-of select="ev:GovTalkMessage/ev:GovTalkDetails/ev:GovTalkErrors/ev:Error/ev:Text"/> </Text> <Location> <xsl:value-of select="ev:GovTalkMessage/ev:GovTalkDetails/ev:GovTalkErrors/ev:Error/ev:Location"/> </Location> <Number> <xsl:value-of select="ev:GovTalkMessage/ev:GovTalkDetails/ev:GovTalkErrors/ev:Error/ev:Number"/> </Number> <Type> <xsl:value-of select="ev:GovTalkMessage/ev:GovTalkDetails/ev:GovTalkErrors/ev:Error/ev:Type"/> </Type> </GovTalkStatus> </xsl:template> </xsl:stylesheet>   Only thing different about previous XSL files is the references to two namespaces ev & gt. These are defined in the GovTalk response at the top of the response; xsi:schemaLocation="http://www.govtalk.gov.uk/CM/envelope http://xmlgw.companieshouse.gov.uk/v1-0/schema/Egov_ch-v2-0.xsd" xmlns="http://www.govtalk.gov.uk/CM/envelope" xmlns:dsig="http://www.w3.org/2000/09/xmldsig#" xmlns:gt="http://www.govtalk.gov.uk/schemas/govtalk/core" If we do not put these references into the XSLT template then  the XslCompiledTransform object will not be able to find the relevant tags. Deserialization is a fairly simple activity. encoder = new ASCIIEncoding(); ms = new MemoryStream(encoder.GetBytes(statusXML)); serializer = new XmlSerializer(typeof(GovTalkStatus)); xmlTextWriter = new XmlTextWriter(ms, Encoding.ASCII); messageStatus = (GovTalkStatus)serializer.Deserialize(ms);   We set up a serialization object using the object type containing the error state and pass to it the results of a transformation between the XSLT above and the GovTalk response. Now we have an object containing any error state, and the error message. All we need to do is check the status. If there is an error then we can flag an error. If not then  we extract the results and pass that as an object back to the calling function. We go this by guess what – defining an XSLT template for the result and using that to create an Xml Stream which can be deserialized into a .Net object. In this instance the XSLT to create the result of a Company Number Search is; <?xml version="1.0" encoding="us-ascii"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:ev="http://www.govtalk.gov.uk/CM/envelope" xmlns:sch="http://xmlgw.companieshouse.gov.uk/v1-0/schema" exclude-result-prefixes="ev"> <xsl:template match="/"> <CompanySearchResult xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <CompanyNumber> <xsl:value-of select="ev:GovTalkMessage/ev:Body/sch:NumberSearch/sch:CoSearchItem/sch:CompanyNumber"/> </CompanyNumber> <CompanyName> <xsl:value-of select="ev:GovTalkMessage/ev:Body/sch:NumberSearch/sch:CoSearchItem/sch:CompanyName"/> </CompanyName> </CompanySearchResult> </xsl:template> </xsl:stylesheet> and the object definition is; using System; using System.Collections.Generic; using System.Linq; using System.Web; namespace CompanyHub.Services { public class CompanySearchResult { public CompanySearchResult() { CompanyNumber = String.Empty; CompanyName = String.Empty; } public String CompanyNumber { get; set; } public String CompanyName { get; set; } } } Our entire code to make calls to send a request, and interpret the results are; String request = String.Empty; String response = String.Empty; GovTalkStatus status = null; fault = null; try { using (CompanyNumberSearchRequest requestObj = new CompanyNumberSearchRequest()) { requestObj.PartialCompanyNumber = CompanyNumber; request = Toolbox.CreateRequest(requestObj, "CompanyNumberSearch.xsl"); response = Toolbox.SendGovTalkRequest(request); status = Toolbox.GetMessageStatus(response); if (status.Status.ToLower() == "error") { fault = new HubFault() { Message = status.Text }; } else { Object obj = Toolbox.GetGovTalkResponse(response, "CompanyNumberSearchResult.xsl", typeof(CompanySearchResult)); } } } catch (FaultException<ArgumentException> ex) { fault = new HubFault() { FaultType = ex.Detail.GetType().FullName, Message = ex.Detail.Message }; } catch (System.Exception ex) { fault = new HubFault() { FaultType = ex.GetType().FullName, Message = ex.Message }; } finally { } Wrap up So there we have it – a reusable set of functions to send and interpret XML results from an internet based service. The code is reusable with a little change with any service which uses XML as a transport mechanism – and as for the Companies House GovTalk service all I need to do is create various objects for the result and message sent and the relevent XSLT files. I might need minor changes for other services but something like 70-90% will be exactly the same.

    Read the article

< Previous Page | 4 5 6 7 8