Search Results

Search found 149 results on 6 pages for 'crawlers'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 3/6 | < Previous Page | 1 2 3 4 5 6  | Next Page >

  • Will I get penalised by google for overflow hidden? [duplicate]

    - by Rascal Capac
    This question already has an answer here: How bad is it to use display: none in CSS? 5 answers I have 2 divs next to each other, div A and div B. Div A displays a whole wordpress post and div B displays about 15 excerpts of posts with the same tag. To get this looking nice I set the hight of div B to be the same as the hight of div A with javaScript. This means alot of post excerpts will be hidden due to overflow hidden on div B. Will Google penalise me for this, and if so, how can I make this work and still please Google? Ps. I would gladly prevent the excerpts from being indexed but the main post must be indexed Ds.

    Read the article

  • How long is the penalty for Duplicate ecommerce content after it has been ressurected

    - by will
    I am fixing all of the duplicate content on my ecommerce site with all orignal descriptions etc. How long does it take google to start ranking it again? I used to have a good ranking that converted quite a few sales, in the last week i have had next to nothing. Also would the disclaimer i created under each product be considered duplicate content because it is on most of my product pages & is the same.

    Read the article

  • If C-Panel Indexing Manager sets a folder to "No Indexing" can it be crawled by a webcrawler?

    - by Graham
    People are able to view directories / folders on my site right now. So, they could go to mysite.com/images and see the full index. To prevent this, C-Panel offers an option to set a directory / folder to "No Indexing" under the "Index Manager." Will this option allow webcrawlers to crawl / index the images? Or, is there a simpler alternative to block access to all folders directly while still having it SEO friendly? My old server restricted direct access to folders by default. But, the new one does not. Any ideas on this? Thanks!

    Read the article

  • How do I make the home page of the website to come up in the rankings than the internal pages? [closed]

    - by Shahab
    Possible Duplicate: What are the best ways to increase your site's position in Google? Suppose I have a website, e.g. www.example.com that comes at number 6 on the Google search rankings. But the internal pages of the website i.e. www.example.com/index.php?a=1&b=2 or www.example.com/index.php comes at number 2 of the rankings. How would I make my prime domain name www.example.com to come at the top of the list ? Any guidance would be appreciated.

    Read the article

  • Does searching a keyword on Google make the crawlers look harder in the future?

    - by Foo Bar
    Do the search requests made by the users influence the Google crawlers "attraction" by this keyword? Let's say Google has some hits on a specific keyword in the search index. And now I search for exactly this keyword. Will the Google crawlers react to the search and keep looking more intense for pages that could match this keyword? A reason why this could be important: Privacy when searching yourself. Assume you just want to know how much Google (and thus other people) can find out about you. If now any (statistical) additional search for your name trigger the crawlers even one step harder to find even more about you, it would have the negative effect that you would actually be found easier in the future, even though you had the intention and hope to find out how few Google finds about you. It's a bit like the dillema in quantum mechanis: Does observing the system automatically change the system?

    Read the article

  • SEOs: mobile version using AJAX: how to be properly read by crawlers?

    - by Olivier Pons
    Before anything else, I'd like to emphasize that I've already read this and this. Here's what I can do: First choice: create classical web version with all products in that page - http://www.myweb.com. create mobile web version with all products in the page and use jQuery Mobile to format all nicely but this may be long to (load + format), and may provide bad user experience - http://m.myweb.com. Second choice: create classical web version with all products in that page create mobile web version with almost nothing but a Web page showing wait, then download all products in the page using AJAX and use jQuery Mobile to format all nicely. Showing a wait, loading message gives far more time to do whatever I want and may provide better user experience - http://m.myweb.com. Question: if I choose the second solution, Google won't read anything on the mobile version (because all products will be downloaded in the page using AJAX), so it wont be properly read by crawlers. What / how shall I do?

    Read the article

  • How to protect SHTML pages from crawlers/spiders/scrapers?

    - by Adam Lynch
    I have A LOT of SHTML pages I want to protect from crawlers, spiders & scrapers. I understand the limitations of SSIs. An implementation of the following can be suggested in conjunction with any technology/technologies you wish: The idea is that if you request too many pages too fast you're added to a blacklist for 24 hrs and shown a captcha instead of content, upon every page you request. If you enter the captcha correctly you've removed from the blacklist. There is a whitelist so GoogleBot, etc. will never get blocked. Which is the best/easiest way to implement this idea? Server = IIS Cleaning out the old tuples from a DB every 24 hrs is easily done so no need to explain that.

    Read the article

  • 302 vs 301 redirect in this specific case

    - by Binder
    We have a website that displays information in a location based manner, i.e. it detects the IP of the visiting user and redirects him/her to an appropriate landing page; for e.g. a user coming from 'Egypt' will be redirected to http://www.mysite.com/egypt/cairo and a user visting from dubai will be redirected to http://www.mysite.com/uae/dubai, so on and so forth and we cater to multiple locations in the middle-east. Now, we have been advised by our SEO consultant that we should put a 301 (permanent redirect) on http://www.mysite.com to point to http://www.mysite.com/ksa/riyadh I would like to know the negative implications that this would have on Google indexing or otherwise, as I fundamentally disagree with this suggestion and believe that in a siutation like this a 302 redirect would be more appropriate.

    Read the article

  • If an visitors IP address contains "google" or a similar keyword, does this mean they were a crawler?

    - by Roscoe
    Hi, I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, yahoo, bot, crawler, etc. Does this mean that those users were infact search engine crawlers? If so, why are their so many crawlers in my visitor records in comparison to genuine human visitors? (and if not what's the explanation?) Thanks in advance.

    Read the article

  • ASP.NET MVC Head Verb and Selenium RC

    - by Rob
    Selenium (RC) is being used to test an ASP.NET 1.1 site. When we make a request via Selenium RC (and then in turn via Firefox or other configured browser) the http verb is "HEAD". We have several form action methods that have separate POST and GET methods decorated with AcceptVerbs(HttpVerbs.Get). These methods are returning a 404 and logging a "a public action method could not be found" error message. Questions: When writing separate Get/Post action methods what is the best practice for handling the Head verb? Should we always decorate with an AcceptVerbs(HttpVerbs.Get | HttpVerbs.Head)? The only time we've seen HEAD verbs is from Selenium and from some crawlers. We created robots.txt entries for the crawlers. Does anyone know why HEAD verbs are issued via Selenium (instead of an If-Modified-Since header)? Are there mainstream crawlers that use the HEAD verb? Does it affect SEO ranking to not have it?

    Read the article

  • The Sitemap Paradox

    - by Jeff Atwood
    We use a sitemap on Stack Overflow, but I have mixed feelings about it. Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site. Based on our two years' experience with sitemaps, there's something fundamentally paradoxical about the sitemap: Sitemaps are intended for sites that are hard to crawl properly. If Google can't successfully crawl your site to find a link, but is able to find it in the sitemap it gives the sitemap link no weight and will not index it! That's the sitemap paradox -- if your site isn't being properly crawled (for whatever reason), using a sitemap will not help you! Google goes out of their way to make no sitemap guarantees: "We cannot make any predictions or guarantees about when or if your URLs will be crawled or added to our index" citation "We don't guarantee that we'll crawl or index all of your URLs. For example, we won't crawl or index image URLs contained in your Sitemap." citation "submitting a Sitemap doesn't guarantee that all pages of your site will be crawled or included in our search results" citation Given that links found in sitemaps are merely recommendations, whereas links found on your own website proper are considered canonical ... it seems the only logical thing to do is avoid having a sitemap and make damn sure that Google and any other search engine can properly spider your site using the plain old standard web pages everyone else sees. By the time you have done that, and are getting spidered nice and thoroughly so Google can see that your own site links to these pages, and would be willing to crawl the links -- uh, why do we need a sitemap, again? The sitemap can be actively harmful, because it distracts you from ensuring that search engine spiders are able to successfully crawl your whole site. "Oh, it doesn't matter if the crawler can see it, we'll just slap those links in the sitemap!" Reality is quite the opposite in our experience. That seems more than a little ironic considering sitemaps were intended for sites that have a very deep collection of links or complex UI that may be hard to spider. In our experience, the sitemap does not help, because if Google can't find the link on your site proper, it won't index it from the sitemap anyway. We've seen this proven time and time again with Stack Overflow questions. Am I wrong? Do sitemaps make sense, and we're somehow just using them incorrectly?

    Read the article

  • Is my robots.txt working as it should?

    - by TigerBlood
    I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in google search results, but I am not coming up in Bing, Yahoo, etc. I have had the same robots.txt since last year, and I initially requested inclusion ~1 year ago, having also resubmitted the URL to those latter search engines several times since as well. Is my robots.txt blocking those other crawlers? And if so, why not google as well? Thanks in advance!

    Read the article

  • web spidering/crawling, can i do it or just search engines?

    - by bboyreason
    i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able to get specific data like links or, in my case, products. all of the products on my site have the following naming convention, website.com/uniqueAlphaNumericID.html as far as i know, no dynamic content generation is being used and only one page per one item in the above format. should i just be thinking about: wget website.com | grep *.html or should i be looking into spiders/crawlers?

    Read the article

  • Sticky Load Balancing with AWS

    - by John Wheal
    I have just setup a load balancer with AWS for a few instances as search engine crawlers were bringing down the site (it has millions of pages). Parts of the site allow you to login so I selected: Enable Application Generated Cookie Stickiness and everything works fine. I now wonder how this will effect my SEO and the crawlers. As I selected sticky load balancing does this mean that a crawler will be stuck on one server and therefore defeat the point in the load balancer? Any recommendations will be appreciated.

    Read the article

  • How should I deal with user agent parsing in logs?

    - by Mr. Jefferson
    My web app project includes logging functionality so we can see where visitors are coming from (referrer URL), what the popular user agents are, what pages are most popular, etc. The log is stored in SQL Server, and when I query the user agents I use a large (almost 100 lines) and growing CASE statement to separate the user agents using string matching (i.e. if the user agent contains the string "Firefox/9" then it's Firefox 9). Is there a better way to do this so I don't have to continually add to that CASE statement to deal with new browser releases? Also, how should I deal with less common, weird/unknown user agents? I've seen the following in the logs and been unable to find good information online about what they are: WordPress/3.3.1; http://www.facecolony.org Mozilla/4.0 ( http://www.hairirons.org redips; <a href=http://hairirons.org/>chi hair iron</a>) I'd guess they're bots/crawlers, but the sites they point to don't appear to reference web crawlers (or even be available sometimes). I've seen other user agents aren't familiar to me, but I know they're bots because they include "bot" or "spider" or something similar in them.

    Read the article

  • Searchengine bots and meta refresh for disabled Javascript

    - by Jonathan
    Hi! I have a website that must have javascript turned on so it can work there is a < noscript tag that have a meta to redirect the user to a page that alerts him about the disabled javascript... I am wondering, is this a bad thing for search engine crawlers? Because I send an e-mail to myself when someone doesn't have js so I can analyze if its necessary to rebuild the website for these people, but its 100% js activated and the only ones that doesn't have JS are searchengines crawlers... I guess google, yahoo etc doesn't take the meta refresh seriously when inside a < noscript ? Should I do something to check if they are bots and do not redirect them with meta? Thanks, Joe

    Read the article

  • Extracting data from internet

    - by Ankiov Spetsnaz
    I would like to extract data from internet like www.mozenda.com does but I want to write my own program to do that. Specific data I'm looking for is various event data. Based on my research, I think custom web crawler is my answer but I Would like to confirm the answer and see if there are any suggestion to make custom web crawlers if web crawler indeed is an answer. Personally, I would prefer Java and I'm planning on using Glassfish technology if that matters...

    Read the article

  • Online SEO Tool to Analyze Your Website

    Learning search engine optimization can be difficult without some sort of SEO tool to help decipher in which direction your marketing efforts are headed. Though there are, in fact, many online SEO tools that can help with your internet marketing campaign, perhaps the most beneficial is an SEO tool that analyzes your entire website. This type of tool, usually offered free of charge, acts in a similar way to search engine crawlers.

    Read the article

  • Fetch as Google error 403

    - by Bojan Vidanovic
    2 weeks ago, google cant access my website anymore, in webmaster tools i cant fetch any page, i always get error 403, and the website has been completly disapperard form the google search results. I cant figure how suddendly it cant see it anymore, i've checked .htaccess and there nothing that blocks google crawlers, and robots.txt is fine to. Anyway the site is accesibly normaly for users. Anyone had this problems? please help!

    Read the article

  • SEO Tips

    SEO is the acronym for search engine optimization. The term "optimization" has been defined as a complete set of procedures that are implemented to make the system fully functional. In the case of search engine optimization, the websites are designed to match the algorithm of the search engines so that the "crawlers" can easily pick up the pages for indexing purposes.

    Read the article

  • Using CSS3 is a bad practice? [closed]

    - by Qmal
    Possible Duplicate: Should I use HTML5 and/or CSS3 to build my website? I just want to know if it's considered as a "bad practice" to use things like rounded corners, gradients and so on... I understand that there are bots and crawlers that do not process CSS, but they don't need to. And nowadays most people use browsers that can process CSS3 with no problem. So should I make my buttons and shadows and such look pretty with CSS3 or with images?

    Read the article

  • Inputting Visual Extras in Your Webpage and SEO

    If you or your SEO consultant plan on having a successful website that is ranked high and recognized by search engines and crawlers, make sure that you review the below points to different visual extras that may be put into your website and possibly affect its rating. It's good Search Engine Optimization to have an attractive website, however, too many, and in some cases, any of these visual extras may actually hurt your website.

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 1 2 3 4 5 6  | Next Page >