When the error occured on my website, Wordpress 3.5 was up and running. So I'm still tending to point at the execution time.
Perhaps Yoast is capturing the entries for the sitemap in a different, far more execution time expensive way than the other plugins?
@crashguru: Did you try my suggestion reducing the number of entries in Yoast? I'm still pretty convinced of this as a possible solution.