Social Media Impact: Well-known traits on social media can spill over into search engines like yahoo, impacting the search quantity of relevant keyword phrases. txt file is then parsed and may instruct the robotic as to which webpages will not be to be crawled. To be a search engine crawler https://www.youtube.com/watch?v=DhNbKpYq6Ro