Cross Domain Video and Image XML sitemaps with a CDN
The Google Sitemap protocol enables you to provide details about your pages to search engines, sitemaps provide additional information about site pages beyond just the URLs. Typically, it is best practice to submit XML sitemaps for pages, images and videos. In the case of a CDN there are additional steps needed to implement XML sitemap submission.
Unlike regular single server websites, images and/or videos are typically hosted across multiple servers. Generating and submitting XML sitemaps for images and videos on a CDN requires making modifications to robots.txt files on each server.
To submit Sitemaps for multiple hosts from a single host, you need to “prove” ownership of the host(s) for which URLs are being submitted in a Sitemap.
Example: To submit Sitemaps for 3 hosts:
www.host1.com with Sitemap file sitemap-host1.xml
www.host2.com with Sitemap file sitemap-host2.xml
www.host3.com with Sitemap file sitemap-host3.xml
Moreover, you want to place all three Sitemaps on a single host: www.sitemaphost.com. So the Sitemap URLs will be:
By default, this will result in a “cross submission” error since you are trying to submit URLs for www.host1.com through a Sitemap that is hosted on www.sitemaphost.com (and same for the other two hosts). One way to avoid the error is to prove that you own (i.e. have the authority to modify files) www.host1.com. You can do this by modifying the robots.txt file on www.host1.com to point to the Sitemap on www.sitemaphost.com.
In this example, the robots.txt file at http://www.host1.com/robots.txt would contain the line “Sitemap: http://www.sitemaphost.com/sitemap-host1.xml”.
By modifying the robots.txt file on www.host1.com and having it point to the Sitemap on www.sitemaphost.com, you have implicitly proven that you own www.host1.com. In other words, whoever controls the robots.txt file on www.host1.com trusts the Sitemap at http://www.sitemaphost.com/sitemap-host1.xml to contain URLs for www.host1.com. The same process can be repeated for the other two hosts. Finally, submit the Sitemaps from www.sitemaphost.com. The same process can/should be repeated for the other two hosts.
Finally, the Sitemaps from www.sitemaphost.com will need to be submitted to the Search Engines. This is typically done in Webmaster Tools.
For a step by step instruction guide on how to get you cross domain content indexed properly please see: Indexing content across domains step by step guide.
If your company struggles with cross domain content indexing due to your content delivery network (CDN). Give our consulting team a call today!