http://yoast.com/wordpress/seo/faq/#the-seo-title-output-for-the-plugin-doesnt-work-as-expected
In my theme's header.php I changed the current <title>STUFF HERE</title> to: <title><?php wp_title(''); ?></title> and now it's working fine. I know a bunch of people are having the same issue, so here's a fix.