Search Results

Search found 42868 results on 1715 pages for 'web crawlers'.

Page 36/1715 | < Previous Page | 32 33 34 35 36 37 38 39 40 41 42 43  | Next Page >

  • Is this Anti-Scraping technique viable with Crawl-Delay?

    - by skibulk
    I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for users that access an abnormal number of pages per minute. I don't want search engine spiders to ever receive the error. My inclination is to set a robots.txt crawl-delay which will ensure spiders access a number of pages per minute under my 503 threshold. Is this an appropriate solution? Do all major search engines support the directive? Could it negatively affect SEO? Are there any other solutions or recommendations?

    Read the article

  • How to set up offline manifest for a web app to run in Safari in iOS?

    - by ahmd1
    I'm currently trying to set up an offline.manifest file for my web app to be used offline on an iOS device. For testing purposes I have a very simple HTML page that I'm trying to add to a home screen. I'm testing it on a live iPhone 4, but after the page is added to the home screen and I put the iPhone in the airplane mode and try to start my web app I get this error: "Turn Off Airplane Mode or Use Wi-Fi to Access Data" and then if I click OK I get: "Cannot Open Web App Name" "Web App Name could not be opened because it is not connected to the Internet" The following is added to the HTML file: <!DOCTYPE html> <html lang="en" manifest="scrts/offline.manifest"> and the offline.manifest is composed as such: CACHE MANIFEST ../pics/bkgnd_iphn_settings.png ../pics/mbl_btn_fb.png ../pics/mbl_btn_twt.png ../pics/icon_57_57_bg.png ../pics/icon_72_72_bg.png ../pics/icon_114_114_bg.png ../pics/icon_144_144_bg.png ../pics/splash_320_460_bg.png ../pics/splash_768_1004_bg.png ../pics/splash_1004_768_bg.png I got all instructions on composing it from here I also adjusted the .htaccess file to add this line: AddType text/cache-manifest .manifest Any idea what am I not doing right?

    Read the article

  • What technical details should a programmer of a web application consider before making the site public?

    - by Joel Coehoorn
    What things should a programmer implementing the technical details of a web application consider before making the site public? If Jeff Atwood can forget about HttpOnly cookies, sitemaps, and cross-site request forgeries all in the same site, what important thing could I be forgetting as well? I'm thinking about this from a web developer's perspective, such that someone else is creating the actual design and content for the site. So while usability and content may be more important than the platform, you the programmer have little say in that. What you do need to worry about is that your implementation of the platform is stable, performs well, is secure, and meets any other business goals (like not cost too much, take too long to build, and rank as well with Google as the content supports). Think of this from the perspective of a developer who's done some work for intranet-type applications in a fairly trusted environment, and is about to have his first shot and putting out a potentially popular site for the entire big bad world wide web. Also, I'm looking for something more specific than just a vague "web standards" response. I mean, HTML, JavaScript, and CSS over HTTP are pretty much a given, especially when I've already specified that you're a professional web developer. So going beyond that, Which standards? In what circumstances, and why? Provide a link to the standard's specification.

    Read the article

  • What language available on commodity web hosts would suit a C# developer? [closed]

    - by billpg
    Recognising its ubiquity on commodity web hosting services, I tried developing in PHP a few years ago. I really didn't like it, later deciding that life was too short for PHP. (In brief, having to put $ on variable names; mis-spelt variable names become new variables; converting non-numeric strings to integers without complaint; the need for an "and this time I mean it" comparison operator.) In my ideal world, commodity web hosts would all support C#/ASP.NET, my preferred web-development language and framework, but this is not my ideal world. Even Mono has barely made a dent on Linux based hosts. However, last time I moaned about PHP's ubiquity, someone followed up that this was no longer the case, and that many other languages are now commonly usable on web hosts too. What programming language; a. Would suit a developer who prefers C#. b. Is available to run on many web hosts.

    Read the article

  • robots.txt dissalow url containing string with a '/' at the end

    - by thanili
    i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page for article itemA belonging to category catA/subcatA, with URL: /catA/subcatA/itemA this is the URL that i want to be indexed from google. this article is also visible via tagging in various other places in the web site. The URLs produced via tagging is like: /tagA1/itemA this URL i want NOT to be indexed from google. However i want to have indexed all tag listings: /tagA1 so how can i achieve this? dissalow URLs of including a specific string with a '/' at the end? /tagA1/ itemA - dissalow /tagA1 - allow

    Read the article

  • How to build a web service to detect content change(s) at an external website?

    - by Global nomad
    I'm researching ways to build a web service to periodically traverse a predetermined list of web pages (of another external website) to detect if a page's content has changed from editing of the page, and deletion of the page. The end goal is to have this web service post push-notification events to mobile devices. FYI, I've searched and read "Questions with similar titles" here. Thank you for sharing your answers.

    Read the article

  • Are there any conceptual drawbacks to building a web app with C++ and MySQL?

    - by jnthnclrk
    I've inherited a pretty interesting project where there is a good opportunity to take an existing piece of software and turn it into a SaaS web app. As the project is inherited, the code base/ framework is already defined as C++ and MySQL. The app itself is compiled and run as an EXE on Windows Server. The UI is web based and the app works as a kind of server. From what I do know of modern web apps, this is perhaps an unusual choice. These days, most people seem to opt for a PHP framework or Ruby on Rails. Certainly that is the impression I get from reading blogs on the subject. So, I'm very interested to know if a C++ EXE backed by MySQL is a solid foundation for a web app, or whether we should be looking to build in another way?

    Read the article

  • Should my web app have its own domain name?

    - by Daniero
    I'm going to get a domain name for my personal web page. It will contain my blog, photos and other personal stuff, plus different web apps and tools that I'm working on. I have put quite a lot of work in one of the web apps, and I think it has great potential. It covers a niche in a way that no other page has done before and I can see great possibilities to expand it. Via links to webstores and ads I think it could even make some money, and if I play it right this could be the place to go when you're into the specific niche that it covers (sorry for being so vague). My question is, would it be better for a (potentially great) web app like this to have its own domain name (nameofmyapp.com) instead of being a subpage of my personal page (mypersonaldomain.com/nameofmyapp)? Is the web app more likely to be found by others, via Google etc, with its own domain name? Could a "subpage" give more attention (visitors) to my personal page? pros and cons?

    Read the article

  • .Net Intermittent System.Web.Services.Protocols.SoapHeaderException

    - by ScottE
    We have a .net 3.5 web app that consumes third party web services. The proxy was created by adding a web reference to their wsdl. This proxy is not compiled. Our error logging is picking up frequent but intermittent exceptions: An exception of type 'System.Web.Services.Protocols.SoapHeaderException' occurred and was caught If I follow the url to the page that generated the exception, I can't recreate it. Edit: Here is most of the exception - where it bubbled up from Message : Internal Error Type : System.Web.Services.Protocols.SoapHeaderException, System.Web.Services, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a Source : System.Web.Services Help link : Actor : Code : http://schemas.xmlsoap.org/soap/envelope/:Client Detail : Lang : Node : Role : SubCode : Data : System.Collections.ListDictionaryInternal TargetSite : System.Object[] ReadResponse(System.Web.Services.Protocols.SoapClientMessage, System.Net.WebResponse, System.IO.Stream, Boolean) Stack Trace : at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at Vendor.getSearch(getSearchRequest getSearchRequest) in c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\be43c34e\b09edc7e\App_WebReferences.pww-cf-q.0.cs:line 73 Edit 2: Inner exceptions: I sometimes get the following inner exceptions logged: Message : Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. Type : System.IO.IOException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 Source : System Help link : Data : System.Collections.ListDictionaryInternal TargetSite : Int32 Read(Byte[], Int32, Int32) Stack Trace : at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.TlsStream.CallProcessAuthentication(Object state) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.ConnectStream.WriteHeaders(Boolean async) And/Or: Message : An existing connection was forcibly closed by the remote host Type : System.Net.Sockets.SocketException, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 Source : System Help link : ErrorCode : 10054 SocketErrorCode : ConnectionReset NativeErrorCode : 10054 Data : System.Collections.ListDictionaryInternal TargetSite : Int32 Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags) Stack Trace : at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) Update We're still working on it. Originally there was a route issue, which was resolved. We're still getting the inner exception with socket errors. We had MS support involved today, and they looked at some traces and network captures. The web service host does round-robin DNS, and they may be responding on a different IP address for the syn syn/ack from one ip, and the next from a different ip. This is not good. This is likely quite specific to our situation, but perhaps it applies to others as well. Microsoft Network Monitor and an application trace got us the information we needed.

    Read the article

  • What web frameworks and languages have the lowest development time?

    - by waitinforatrain
    Hi guys, I'm trying to get a few web development ideas off the ground. Unfortunately I will be the only developer, so I'm focusing on getting a good debuggable, testable setup going that I can develop applications rapidly with. Which languages, development styles and frameworks would you recommend for rapid development? Last time I checked RoR was the next big thing but that was a while back.

    Read the article

  • IE8 web slice - CSS problem

    - by rjovic
    Again me with IE8 web slice :) This time i have following problem. I created web slice for IE8 and created button for users where they can choice to add it to their browsers. Code is : <input class="add" type="button" value="Dodaj Xica web slice u IE8!" onclick='window.external.AddToFavoritesBar("http://localhost:51914/Home/GetWebSlice", "xica.rjovic.com", "slice");' /> There is everything fine, and web slice is added to IE8 and web page is displayed correct. But I have problem with CSS in web slice. I defined css in it, but when web slice i rendered there are only black fonts without any css properties. My web slice is defined as : <%@ Page Language="C#" Inherits="System.Web.Mvc.ViewPage" %> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" > <head runat="server"> <title>GetWebSlice</title> <style type="text/css"> p {color: white; } body {background-color: blue; } .container {background-color: red;} </style> </head> <body> <div class="hslice container" id="xica"> <h2 class="entry-title container">Xica web slice</h2> <div class="entry-content"> <h2>Saldo : <b><%= ViewData["total"] %></b></h2> <p><%= ViewData["cardNumber"] %></p> <p>Status : <%= ViewData["status"] %></p> </div> </div> </body> </html> When I try to see web slice directly in my browser then everything is ok, and css is show as expected. Thank you...! EDIT : here is the picture of problem : http://www.deviantpics.com/share-2F10_4BD73E25.html

    Read the article

  • How to develop asp.net web service to create the web method which can take the parameter of type win

    - by Shailesh Jaiswal
    I am developing asp.net web service. I am developing this web service so that OPC ( OLE for process control) client application can use it. In this web service I am using the built-in functions provided by the namespaces using OPC, using OPCDA, using OPCDA.NET. I have also added the namespace using System.Windows.Forms in this web service so that I can use the windows form control. In that we service I have created on web method which takes the parameter of type windows form control as given below. public void getOPCServerItems(TreeView tvServerItems, ListView lvBranchItems) { ArrayList ArrlstObj = new ArrayList(); ItemShowTreeList = OpcSrv.ShowBrowseTreeList(tvServerItems, lvBranchItems); ItemShowTreeList.BrowseModeOneLevel = true; // browse hierachy levels when selected. (default) ItemShowTreeList.Show(OpcSrv.ServerName); } In the above web method I need to pass the values to the built-in function ShowBrowseTreeList() (found in OPC, OPCDA, OPCDA.NET namespaces). This function takes the two parameter of windows form control type. These parameters are Treeview & ListView control of the windows form. In the above web method ShowBrowseTreeList() method automatically create the treeview & listview structure of the available items. Now as I am consuming the web service so I need to pass the values to the webmethod getOPCServerItems(). But as I my consuming application is asp.net application there is no such windows form control. In asp.net application there are also & control. I want to display The data returned in these asp.net controls rather than windows form control. I am not getting the way what should I need to do or how should I pass the values form my client application to this web service ? In the above method getOPCServerItems() when I use the parameter of type treeview & listview it generate s error "Cannot serialize member System.ComponentModel.Component.Site of type System.ComponentModel.ISite because it is an interface.". Can you provide me the the way In which I can write the above web method & how should I pass parameter to the Treeview & Listview control (windows form control) from my asp.net application ? which controls I should use to pass parameters ? Is there any need to do any type of casting ? Can you provide me the the code for above web method so that I can resolve the above issue ?

    Read the article

  • A good open source web crawler for indexing Specific website for specific contents?

    - by Peeyush
    Hello Please suggest me a good open source web crawler written in C++,JAVA or PHP. i just need to crawl/index some specific websites for specific contents(images,text,videos). i know that their are already a lot of question & answers about this topic on this website but i am a little confused after reading all of them. So i am sorry if i am repeating the same question again. -Thanks in advance

    Read the article

  • What's the easiest and safest way to record data being inputted by a user on a web site

    - by fred august
    Apologies, this is a tragically simple question that will bore most of you. I need to implement the simplest "leave your email and we'll contact you" web page. The simplest thing I could think of is doing an HTML form which calls a PHP script which appends the data in some file on the server. Easy to implement, but now I'm wondering if it's totally hackable. Is it? Are there obvious better ways that are still simple? thanks f

    Read the article

  • How can I setup a flexible local web development environment I can easily sync with a production, Amazon AWS based environment?

    - by user607057
    I'm running on an OS X environment and would like to create a flexible web development environment locally... including the option to run on an Nginx server for my PHP-based application. At the end of the day (or, development cycle), I'd like to be able to hit a switch and have it all sync over to Amazon AWS hosting (EC2, S3) - instances, databases, files, configurations, and all. Are there any simple ways to do this?

    Read the article

  • "NT AUTHORITY\ANONYMOUS LOGON" error in Windows 7 (ASP.NET & Web Service)

    - by Tony_Henrich
    I have an asp.net web app which works fine in Windows XP machine in a domain. I am porting it to a Windows 7 stand alone machine. The app uses a web service which makes a call to sql server. The web server (IIS 7.5) and SQL Server are on the same stand alone machine. I enabled Windows authentication for the website and web service. The web service uses a trusted connection connection string. The web service credentials uses System.Net.CredentialCache.DefaultCredentials. I noticed username, password and domainname are blank after the call! The webservice and web site use an application pool with identity "Network Service". I am getting an exception "NT AUTHORITY\ANONYMOUS LOGON" in the database call in the web service. I am assuming it's related to the blank credentials. I am expecting ASPNET user to be the security token to the database. Why is this not happening? (Usually this happens when sql server and web server are on two different machines in a domain, delegation & double hopping, but in my case everything is on a dev box)

    Read the article

  • The rules to connect a web service trough the SSL and Certificates

    - by blgnklc
    There is a web service running on tomcat on a server. It is built on Java Servlet. It is listening others to call itself on a SSL enabled http port. so its web service adreess looks like: https://172.29.12.12/axis/services/XYZClient?wsdl On the other hand I want to connect the web service above from a windows application which is built on .NET frame work. Finally, when I want to connect the web service from my computer; I get some specific erros; Firstly I get; Proxy authentication error; then I added some new line to my code; Dim cr As System.Net.NetworkCredential = New System.Net.NetworkCredential("xname", "xsurname", "xdomainname") Dim myProxy As New WebProxy("http://mar.xxxyyy.com", True) myProxy.Credentials = cr Secondly, after this modifications It says that bad request. I did not get over this error. Moreover I did try to connect the web server on the same computer. I copied my executable program to the computer where the web service runs. The error was like; The underlying connection was closed: Could not establish trust relationship for SSL/TLS secure channel PS: When I try to connect to web service by using Internet Explorer; I see firstly some warnings about accepting an unknown certificate and I click take me to web service an I get there clearly. I want to know what are the basic elements to connect a web service, could you please tell me the requirements that I have to use on my windows project. regards bk

    Read the article

  • How to Connect to a web service (SSL enabled and proxy interface)

    - by blgnklc
    There is a web service running on tomcat on a server. It is built on Java Servlet. It is listening others to call itself on a SSL enabled http port. so its web service adreess looks like: https://172.29.12.12/axis/services/XYZClient?wsdl On the other hand I want to connect the web service above from a windows application which is built on .NET frame work. Finally, when I want to connect the web service from my computer; I get some specific erros; Firstly I get; Proxy authentication error; then I added some new line to my code; Dim cr As System.Net.NetworkCredential = New System.Net.NetworkCredential("xname", "xsurname", "xdomainname") Dim myProxy As New WebProxy("http://mar.xxxyyy.com", True) myProxy.Credentials = cr Secondly, after this modifications It says that bad request. I did not get over this error. Moreover I did try to connect the web server on the same computer. I copied my executable program to the computer where the web service runs. The error was like; The underlying connection was closed: Could not establish trust relationship for SSL/TLS secure channel PS: When I try to connect to web service by using Internet Explorer; I see firstly some warnings about accepting an unknown certificate and I click take me to web service an I get there clearly. I want to know what are the basic elements to connect a web service, could you please tell me the requirements that I have to use on my windows project. regards bk

    Read the article

< Previous Page | 32 33 34 35 36 37 38 39 40 41 42 43  | Next Page >