\n Stay Forward <\/h2>\n
\n Get exclusive insights into digital
media's top-trending topics delivered
directly to your inbox. <\/p>\n <\/div>\n\n
\n
\n
![\"robots.txt\"](\"http:\/\/www.make-it-rain.co.uk\/wpe-content\/uploads\/2014\/10\/Screenshot-2014-10-28-14.49.50.png\")
<\/a><\/p>\nThis can cause problems if a sites robots.txt accidently blocks JavaScript, CSS and images from being crawled. This can happen when a CMS (content management system) has a default robots.txt that blocks the key files that are required to display a page from Google’s crawler’s gaze. If you do this Google have gone on record<\/a> as saying:<\/p>\n Disallowing crawling of JavaScript or CSS files in your site\u2019s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.\u201d<\/p><\/blockquote>\n This is a major change and indicates that how a site looks to a human user now factors as a ranking metric. After all Google want to provide the end user the best possible sites. And in this day and age a text-based site simply wont do. We have come to expect imagery, video and a host of other points of interaction. A Robots.txt file is a powerful tool, but the downside is that even a minor error in the robots.txt file can cause major disruption. We have seen sites block so much of their JavaScript, images and CSS that they appear to Google as a text-only website. In the past we have seen sites accidentally block themselves entirely, which had a massive impact on the traffic to their site.<\/p>\n\n \n