.Google's John Mueller answered a question regarding why Google marks web pages that are actually forbidden from crawling by robots.txt and also why the it is actually risk-free to overlook the associated Search Console documents concerning those crawls.Crawler Visitor Traffic To Question Parameter URLs.The individual asking the concern recorded that robots were actually creating web links to non-existent concern guideline URLs (? q= xyz) to webpages with noindex meta tags that are additionally shut out in robots.txt. What caused the question is that Google is actually crawling the hyperlinks to those pages, acquiring blocked through robots.txt (without watching a noindex robotics meta tag) at that point getting shown up in Google Search Console as "Indexed, though shut out by robots.txt.".The individual asked the following concern:." Yet here is actually the significant question: why would certainly Google.com index webpages when they can't also see the material? What's the conveniences in that?".Google.com's John Mueller affirmed that if they can not crawl the webpage they can't view the noindex meta tag. He additionally creates an exciting mention of the site: search operator, suggesting to overlook the end results due to the fact that the "average" individuals won't see those end results.He composed:." Yes, you are actually appropriate: if our team can't crawl the webpage, our team can not find the noindex. That stated, if we can not crawl the web pages, after that there's certainly not a great deal for our company to index. So while you may observe some of those pages with a targeted site:- question, the average individual will not observe all of them, so I definitely would not fuss over it. Noindex is likewise alright (without robots.txt disallow), it just implies the URLs will certainly find yourself being actually crept (and also end up in the Search Console document for crawled/not recorded-- neither of these conditions create concerns to the rest of the website). The important part is that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the restrictions in using the Website: search progressed hunt operator for analysis explanations. Among those reasons is due to the fact that it's not linked to the regular search mark, it is actually a different point entirely.Google.com's John Mueller discussed the site search driver in 2021:." The brief response is that an internet site: concern is actually certainly not implied to be total, nor used for diagnostics objectives.An internet site question is actually a specific kind of hunt that restricts the outcomes to a particular web site. It is actually essentially simply words internet site, a digestive tract, and then the site's domain.This inquiry limits the results to a details internet site. It's not indicated to be a thorough assortment of all the web pages coming from that website.".2. Noindex tag without making use of a robots.txt is actually great for these type of scenarios where a bot is linking to non-existent pages that are obtaining found by Googlebot.3. URLs along with the noindex tag are going to create a "crawled/not catalogued" item in Explore Console and that those won't have a damaging impact on the remainder of the web site.Read through the question as well as address on LinkedIn:.Why will Google.com mark webpages when they can not even observe the web content?Included Photo through Shutterstock/Krakenimages. com.