Script-Served Content Indexing

Google has supported crawling and indexing of AJAX content for several years now, and even created special guidelines utilising HTML snapshot capabilities to help webmasters ensure that their rich content served this way was discovered and indexed fully.

But times have changed and the capabilities of Google’s crawlers to view pages almost exactly as end user browsers do, has made specialist crawl mechanisms more or less entirely redundant. Google confirmed in May 2014 that it could execute both JavaScript and CSS, and asked webmasters to ensure that resources weren’t blocked or inaccessible to the Googlebot.

The Test – How to check Script-Served Content Indexing

Being a sceptical and curious lot at 4Ps Labs, we decided to try this out using Google Tag Manager (GTM), which amongst its many other delightful capabilities is able to serve custom HTML content. As GTM is executable JavaScript, if Google was unable to index content served through GTM then we’d know that all this JavaScript execution business was nonsense. If it did, we could get a ballpark comparison of how long it takes to index versus conventional inline HTML content.

Widgetzone Screencap Showing Browser Content Deployment

First, we registered a new domain name and set it up on Google Webmaster Tools. We made two versions of the homepage – one with a charming poem about widgets served via Google Tag Manager custom HTML, and one with a different poem served as normal inline HTML.

The inline HTML page was put up first and tracked using Stat, which takes ranking samples from SERPs based on specific query terms. We were aiming for an immensely long-tailed search term of course for purposes of this experiment, and after WMT submission it only took 3 days for the content to appear in Google results.

Note that this was just the raw content – no links, no social, nothing but a single webpage and a Webmaster Tools account. Not bad, eh?

We then replaced this with the version of the homepage that had different content on it, all deployed this time via Google Tag Manager so the source code of the page looked like this:

Google Tag Manager Content Deployment

This took a little longer – about three days to initially become visible, then a hiccup for over a week before it started ranking consistently for the page H2 term. Note that at no point did the page become visible at all in Bing or Yahoo even after the WMT submission – but then those engines have never claimed to be AJAX crawling experts, which is fair enough. The SERP sampling for these terms is provided by our chums over at Stat, if anyone wants to see visibility graphing like this for themselves.

Widgetzone Ranking JS Deployment

So there you have it – yes, Google does indeed appear to execute JavaScript, although it looks like it might not index it quite as rapidly or smoothly as it does inline HTML content. Some wider experimenting needed here to get some more definitive conclusions – watch this space for more from 4Ps labs, or drop us a tweet #4PsLabs via @4PsMarketing if you have a question, comment or just an idea for an analytics, content or search experiment.