Fixed it by replacing the following code...
$pluginurl = plugin_dir_url( __FILE__ );
with...
$pluginurl = preg_replace( '/https?:\/\/[-a-zA-Z0-9.]+\//', home_url('/'), plugin_dir_url( __FILE__ ) );
in wp-seo.php at line number 59, as of version 1.3.4.4 of this plugin.
The above fix basically lets the wp-content/plugins/wordpress-seo/css/xml-sitemap-xsl.php to be served from the same domain as the site (without using the CDN domain).
Hope this helps someone!