I had a very weird answer from my hosting company (hostgator) and I wanted to double check with you. To cut a long story short, here is a copy paste of their answer regarding a slow first byte on 3 of my sites (I use WordPress SEO on ALL my sites and have no issues there, so I don't think the plugin is causing that):
Quoting hostgator:
The time to first byte issues you are seeing on umbrellight.com, rhetorical-ratatouille.com, and froggystyle.biz, all appear to be due to the plugin configuration taking a large amount of time. While you do have caching enabled on these sites, all of them are using WordPress SEO, which specifically disables caching, so each access of the site does require generation of the page being accessed. Disabling WordPress SEO or replacing it with something like All in One SEO or Greg's High Performance SEO should remedy these problems.
Please advise.