I know this has been suggested but I can't find it... It may even already be to-do.
Anyway, there should be in an game option to start a vote from casual to speed or vice versa. A speed to casual vote should last up to an hour and you need everyone in game to say yes. A casual to speed vote should last 5 minutes and you need everyone in the game to be premium and to say yes.
There are several reasons this is a good idea, but I'm playing a speed game I wish could turn casual right now so I'm too jumpy to type for very long.
Priority 3/5
PS. Err also, the same person can't force this vote twice in a row.