Search Results

Search found 499 results on 20 pages for 'robots'.

Page 6/20 | < Previous Page | 2 3 4 5 6 7 8 9 10 11 12 13  | Next Page >

• How do I deal with content scrapers? [closed]

  - by aem

  Possible Duplicate: How to protect SHTML pages from crawlers/spiders/scrapers? My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling for that name produces various results of people who've concluded that it doesn't respect robots.txt (eg, http://www.0sw.com/archives/96). I'm…

  Read the article

• SEO: disallowing Google from indexing forms in iframes or not?

  - by Marco Demaio

  I usually place forms in iframes (i.e. order form, request assistance form, contact forms, ect.). Just the forms, I never place other contents or pages in iframes. From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases, moreover I obviously…

  Read the article

• CDN virtual subdomain causes duplicated content

  - by user3474818

  I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act as an CDN and serve static content in order to improve speed. However, all of my content can be accessed via subdomain aswell, so Google…

  Read the article

• Googlebot can't access my site when crawling from rootdomain

  - by PéCé

  I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. Here is my site specifics : vide-greniers.trocmalin.com is the site address www.trocmalin.com redirects (301) to…

  Read the article

• Will duplicate international (i18n) content hinder SEO rankings?

  - by Rhys

  Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that is often replicated across locales. For example, a site's /en-us/about page could be identical to /en-uk/about,…

  Read the article

• Search engine bots accessing strange URLs

  - by casasoft

  We have ELMAH enabled on our site and get errors whenever a Page Not Found error is triggered on the website. We have recently redesigned a new website and so we understand that search engine robots might have previously indexed pages which they try to access and result in a Page Not Found errors. For this…

  Read the article

• Htaccess/robots.txt to allow search bots to explore main domain but not directory on other domain

  - by gX

  Ok, I understand the Title didn't make any sense so here I've tried to explain it in detail. I'm using a hosting that gives me space for my domain and lets me "add on" other domains on it. So lets say I have a domain A, and I add on a domain B. Basically my hosting gives me a public_html where I can put…

  Read the article

• Handling SEO for Infinite pages that cause external slow API calls

  - by Noam

  I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are generated and he waits. Considering I cannot pre-create them all, I am trying to figure out the best SEO…

  Read the article

• mod_rewrite and SEO friendliness

  - by John Doe

  My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes. I have a unique, large PHP script, and I use the Apache module mod_rewrite in the .htaccess file to create friendly URLs, for example: RewriteRule ^$…

  Read the article

• What kind of redirect (301 or 302) for an email links tracker?

  - by MaxiWheat

  We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application (https://ourdomain.com/trackingurl?blablabla) which then redirects the email reader to the original URL our…

  Read the article

• Does GoogleBot respect User-agent: *

  - by rkulla

  I blocked a page in robots.txt under User-agent: *, and tried to do a manual removal of that URL from Google's cache in the webmasters tools. Google said it wasn't being blocked in my robots.txt, so I then blocked it specifically under User-agent: GoogleBot and tried removing it again…

  Read the article

• [Disallow: /index.php] seems to block /my-beautiful-sef-url-123

  - by Jaroslav Záruba

  Hello I have robots.txt that looks like this: User-agent: * Disallow: /system/ Disallow: /admin/ Disallow: /index.php The obvious goal has been to prevent all the ugly URLs from being indexed, as they all begin with "/index.php". But for some reason all URLs like…

  Read the article

• I'm getting a "Does not implement IController" error on images and robots.txt in MVC2

  - by blesh

  I'm getting a strange error on my webserver for seemingly every file but the .aspx files. Here is an example. Just replace '/robots.txt' with any .jpg name or .gif or whatever and you'll get the idea: The controller for path '/robots.txt' was not found or does…

  Read the article

• Unit testing Google wave robots in Java

  - by Paul

  Any tips or best practices for unit testing Google Wave robots written in Java? I'm expecting to deploy on AppEngine, if that helps. I'm a fan of TDD but new to both Wave Robots and AppEngine, so I'm hoping to use TDD to help me explore the design space.

  Read the article

• Why deny access to website for msnbot/bingbot?

  - by Quandary

  I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for hacking and spamming. But why are there so many sites recommending to…

  Read the article

• What bots are really worth letting onto a site?

  - by blunders

  Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering if the goal of the site allowing bots is for the potential for the bot to send real traffic back to the site if there is any reason to allow bots…

  Read the article

• How to get search engines to properly index an ajax driven search page

  - by Redtopia

  I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial view does not have any records listed, and there is no interface…

  Read the article

• Using rel=canonical and noindex in a 1-n partners enviroment

  - by Telemako Mako

  We sell a whole site (domain, etc) to partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original, but the one that is indexed is the partners copy. We want to do it this way so the…

  Read the article

• Adsense click bot is click bombing my site

  - by Graham

  I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then de-credited soon after. So, they were obviously fraudulent clicks.…

  Read the article

• Should I block bots from my site and why?

  - by Frank E

  My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and why? Which ones have a legitimate purpose in increasing…

  Read the article

• Receiving requests where absolute URL on page are morphed to relative URLs

  - by Jacob

  In our web pages, we have a hyperlink with an href to an absolute URL: https://some.other.host.com/blah.aspx?var1=val1&var2=val2 For some reason, in our logs, we see a lot of requests to URLs of this format: …

  Read the article

• Yandex frequently replaces page names with ampersands

  - by Guy

  The Yandex spider is a frequent visitor to one of the sites I manage. On ocassion it replaces the page name with two ampersands and a space. So if the page is: /mypage.aspx?param=value then it will try and crawl it…

  Read the article

• Uploading a non-finished website

  - by Daniel

  I have a pretty basic question. I developed a neet little website wich I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be uploaded. Besides that, I…

  Read the article

• How to hide download file from bots? [closed]

  - by CJ7

  Possible Duplicate: How to restrict the download of all files in a folder? I want to make a private file available for download but not use username/password protection. I want to put the file into a directory…

  Read the article

• Should I set NOINDEX header for my JS, CSS and image files?

  - by Yoga

  Are there any harms if my site send NOINDEX headers for all my static assets? For image files, I refer to those valueless, e.g. background images, button images, etc. Update: more background information I have this…

  Read the article

< Previous Page | 2 3 4 5 6 7 8 9 10 11 12 13  | Next Page >