Search Results

Search found 4984 results on 200 pages for 'robots txt'.

Page 4/200 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • Procedural modeling of Robots?

    - by anon
    Procedural techniques is common for texture synthesis, modeling plants, and modeling terrains. However, I've seen very little work on algorithmic construction of robots, which is a bit surprising given how mechanical these systems are. Anyone have a good resource on the algorithmic construction of robots / robotic humanoids? Thanks!

    Read the article

  • Multiple SiteMap: entries in robots.txt?

    - by user306942
    I have been searching around using Google but I can't find an answer to this question. A robots.txt file can contain the following line: Sitemap: http://www.mysite.com/sitemapindex.xml but is it possible to specify MULTIPLE sitemap index files in the robots.txt and have the search engines recognize that and crawl ALL of the sitemaps referenced in each sitemap index file? For example, will this work: Sitemap: http://www.mysite.com/sitemapindex1.xml Sitemap: http://www.mysite.com/sitemapindex2.xml Sitemap: http://www.mysite.com/sitemapindex3.xml

    Read the article

  • mean of robots.txt at yahoo.com

    - by hussain
    i want to know the mean of yahoo robots.txt that website( http://www.yahoo.com/robots.txt ) have the following lines User-agent: * Disallow: /p/ Disallow: /r/ Disallow: /*? i dont know the mean of last line(Disallow: /*?) please let me know... thanks and advance

    Read the article

  • wget not respecting my robots.txt. Is there an interceptor?

    - by Jane Wilkie
    I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little. I have implemented a robots.txt policy. I posted it below.. User-agent: wget Disallow: / User-agent: libwww Disallow: / User-agent: * Disallow: / Issuing a wget from my totally independent ubuntu box shows that wget against my server just doesn't seem to work like so.... http://myserver.com/file.csv Anyway I don't mind people just grabbing the info, I just want to implement some sort of flood control, like a wrapper or an interceptor. Does anyone have a thought about this or could point me in the direction of a resource. I realize that it might not even be possible. Just after some ideas. Janie

    Read the article

  • Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?

    - by CaptainCodeman
    As I understand, having repetitive content is a detriment to search engine placement. Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due to similar legal wording or due to copy & pasting from the same source, would it be a good idea to disallow access to these pages via robots.txt, in order to avoid being penalized for "non-original content"? Or, on the contrary, could the search engines identify this as circumvention and penalize the site for trying to hide content? Or does it not matter?

    Read the article

  • robots.txt, how effective is it and how long does it take?

    - by Stefan
    We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php. When you search the company on engines such as Google, you get the site and a listing of its sub pages which now lead to outdated pages. Our plan doesn't allow us to edit the .htaccess and the old pages are .html docs so I cannot use PHP redirects either. So if I put in place a robots.txt telling the engines to not crawl beyond index.php, how effective will this be in preventing/removing crawled sub pages. And rough guess, how long before the search engines would update?

    Read the article

  • Cross-domain jQuery using YQL gives robots.txt error

    - by Jens Roland
    On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows the JSON result, including the error message: "forbidden":"robots.txt for the domain disallows crawling for url: http://qxlapp.dk/projects/dagens_kup/show.php". The robots.txt on the qxlapp.dk domain contains the following: User-agent: Yahoo Pipes 2.0 Allow: / User-agent: * Allow: / So I don't see what the problem is? Shouldn't it pull the page just fine with those settings?

    Read the article

  • SEO chaos from changing robots.txt file in Wordpress site

    - by Seedorf
    Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase my search ranking by limiting the pages being accessed by google. This is my robots.txt file in wordpress: User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads Sitemap: http://www.instant-wine-cellar.co.uk/wp-content/themes/Wineconcepts/Sitemap.xml

    Read the article

  • read out a txt and send the last line to email adress - vbscript

    - by matthias
    Hallo, I'm back haha :-) so i have the next question and i hope someone can help me... I know i have a lot of questions but i will try to learn vbscript :-) Situation: I try to make a program that check all 5 min a txt and if there a new line in the txt, i'll try to send it to my eMail Address Option Explicit Dim fso, WshShell, Text, Last, objEmail Const folder = "C:\test.txt" Set fso=CreateObject("Scripting.FileSystemObject") Set WshShell = WScript.CreateObject("WScript.Shell") Do Text = Split(fso.OpenTextFile(Datei, 1).ReadAll, vbCrLF) Letzte = Text(UBound(Text)) Set objEmail = CreateObject("CDO.Message") objEmail.From = "[email protected]" objEmail.To = "[email protected]" objEmail.Subject = "Control" objEmail.Textbody = Last objEmail.Configuration.Fields.Item _ ("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 objEmail.Configuration.Fields.Item _ ("http://schemas.microsoft.com/cdo/configuration/smtpserver") = _ "smtpip" objEmail.Configuration.Fields.Item _ ("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25 objEmail.Configuration.Fields.Update objEmail.Send WScript.Sleep 300000 Loop This program works, but this program send me all 5 mins a mail...but i will only have a new mail when there is a new line in the txt. Can someone help me?

    Read the article

  • disallow certain url in robots.txt

    - by chrism
    We implemented a rating system on a site a while back that involves a link to a script. However, with the vast majority of ratings on the site at 3/5 and the ratings very even across 1-5 we're beginning to suspect that search engine crawlers etc. are getting through. The urls used look like this: http://www.thesite.com/path/to/the/page/rate?uid=abcdefghijk&value=3 When we started we add the following to our robots.txt: User-agent: * Disallow: /rate Is this incorrect or are googlebot and others simply ignoring our robots.txt?

    Read the article

  • my robots.txt file in web application

    - by Zerotoinfinite
    Hi, I am using asp.net with C#. To increase the searchibility of my site in google, I have searched & found out that I can do it by using my robots.txt , but I really don't have any idea how to create it and where can I place my tag like 'asp.net, C#' in my txt file. Also, the necessary steps to to include it in my application. Please help. Thanks in advance

    Read the article

  • How can I use varnish to generate a robots.txt file even for subdomain of the same site?

    - by Sam
    I want to generate a robots.txt file using Varnish 2.1. That means that domain.com/robots.txt is served using Varnish and also subdomain.domain.com/robots.txt is also served using Varnish. The robots.txt must be hardcoded into default.vcl file. is that possible? I know Varnish can generate a maintenance page on error. I'm trying to make it generate a robots.txt file. Can anyone help? sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {" <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>Maintenance in progress</title> </head> <body> <h1>Maintenance in progress</h1> </body> </html> "}; return (deliver); }

    Read the article

  • Google Wave Robots API v2

    Google Wave Robots API v2 Pamela Fox describes how Wave Robots works, and new features in Robots API v2. From: GoogleDevelopers Views: 2 0 ratings Time: 17:28 More in Science & Technology

    Read the article

  • Google I/O 2010 - Making smart & scalable Wave robots

    Google I/O 2010 - Making smart & scalable Wave robots Google I/O 2010 - Making smart & scalable Wave robots Wave 201 David Byttow, Marcel Prasetya A smart robot must be able to store persistent data. Wave robots can store data in wave structures, like wavelets, datadocs, and annotations, instead of traditional datastores. A scalable robot must perform operations with minimal bandwidth. Wave robots can optimize by selecting the appropriate amount of context, the optimal events, and narrow filters for events. In this talk, we'll share best practices on data storage and scaling. For all I/O 2010 sessions, please go to code.google.com From: GoogleDevelopers Views: 9 0 ratings Time: 58:25 More in Science & Technology

    Read the article

  • pages still show up in google search even after disallowed in robots.txt [duplicate]

    - by Jota Onasys
    This question already has an answer here: With Robots.txt disallow all, why was my site still getting traffic? 5 answers Why is it that some pages still show up in google search even though disallowed in robots.txt? Is the best solution here to remove the Disallow from Robots.txt and just add noindex, nofollow meta tag to those pages you want blocked? Or should I submit a request to Google directly to remove those pages?

    Read the article

  • TXT File or Database?

    - by Ruth Rettigo
    Hey folks! What should I use in this case (Apache + PHP)? Database or just a TXT file? My priority #1 is speed. Operations Adding new items Reading items Max. 1 000 records Thank you. Database (MySQL) +----------+-----+ | Name | Age | +----------+-----+ | Joshua | 32 | | Thomas | 21 | | James | 34 | | Daniel | 12 | +----------+-----+ TXT file Joshua 32 Thomas 21 James 34 Daniel 12

    Read the article

  • Search for duplicate prefix of file names

    - by Mia
    I have a folder contains files name as: xxx.get.txt and xxx.resp.txt xxx.get.txt, xxx.resp.txt yyy.get.txt, yyy.resp.txt zzz.get.txt, zzz.resp.txt, etc each prefix xxx should have two corresponding files, .get.txt and .resp.txt However, now I calculate the number of .get.txt and the number of .resp.txt files, they are not equal, there're ten more .get.txt. I want to find out, which .get.txt files do not have ´.resp.txt´ file Is it possible? Many thanks!

    Read the article

  • we are getting .txt file but not getting proper alignment

    - by pmms
    we are getting the following texfile_screenshot1.JPG when we are exporting data to .txt file we need output which is shown in texfile_screenshot2.JPG following is the code $myFile = "user_password.txt"; $fh = fopen($myFile, 'a') or die("can't open file"); $newline ="\r\n"; fwrite ($fh,$newline); $stringData1 = $_POST['uname1']." "." "." " ; fwrite($fh, $stringData1); $stringData1 =$_POST['password1']." "." "." "; fwrite($fh,$stringData1); $stringData1 = $_POST['email1']." "." "." "; fwrite($fh, $stringData1); fclose($fh);

    Read the article

  • Conecting bash script with a txt file

    - by cathy
    I have a txt file with 100+ lines called page1.txt; odd lines are urls and even lines are url names. I have a bash file already created that checks urls for completion. Except right now, the process is really manual because I have to modify the bash every time I need to check a url. So I need to connect the bash to the txt using the variable url. $url should get all the odd lines from page1.txt and check if the link is complete or not. Also, how would I write a variable called name that derives from the url the 7 digits? bash file manually: url=http://www.-------------/-/8200233/1/ name=8200233 lynx -dump $url > $name.txt I would prefer if the bash file could add "Complete/In-Complete " at the beginning of every even line in the page1.txt file but a new text file could be also created to keep track of the Completes/In-completes.

    Read the article

  • "X-Robots-Tag: noindex" on an HTTP 301 response

    - by Peter O.
    I understand that a resource with X-Robots-Tag: noindex forces some search engines, including Google, not to index the resource further. I also understand that an HTTP 301 response causes search engines to use the redirected URL instead of the original URL to refer to the resource. But what happens if both "X-Robots-Tag: noindex" and status code 301 occur on the same response? It's likely that the original URL will no longer be indexed, but will that cause the redirected URL to no longer be indexed too? This possibility is not mentioned in the X-Robots-Tag specification.

    Read the article

  • How can robots beat CAPTCHAs?

    - by totymedli
    I have a website e-mail form. I use a custom CAPTCHA to prevent spam from robots. Despite this, I still get spam. Why? How do robots beat the CAPTCHA? Do they use some kind of advanced OCR or just get the solution from where it is stored? How can I prevent this? Should I change to another type of CAPTCHA? I am sure the e-mails are coming from the form, because it is sent from my email-sender that serves the form messages. Also the letter style is the same. For the record, I am using PHP + MySQL, but I'm not searching for a solution to this problem. I was interested in the general situation how the robots beat these technologies. I just told this situation as an example, so you can understand better what I'm asking about.

    Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >