You get the 404 error when the post-sitemap.xml / page-sitemap.xml is empty.
Maybe this will help :
I had Post and Page missing in post-sitemap.xml / page-sitemap.xml. It was the Post/Page having the options "Include in Sitemap" is set to "Auto detect" AND "Meta Robots Index" is set to "Index"
You may change your post/page options this way :
"Include in Sitemap" set to "Always include" should work.
or
"Meta Robots Index" set to "Default for Post type, currently : Index" should work too.
If you don't want to change all posts/pages you can modify the plugin file inc/class-sitemaps.php : what works for me is changing the line 326 for the plugin version 1.3.4.4 (line 325 for the plugin version 1.4.4)
from
if ( wpseo_get_value( 'meta-robots-noindex', $p->ID ) && wpseo_get_value( 'sitemap-include', $p->ID ) != 'always' )
to
if ( wpseo_get_value( 'meta-robots-noindex', $p->ID ) == 1 && wpseo_get_value( 'sitemap-include', $p->ID ) != 'always' )