I've always thought that it might be easiest to have a fixed time at the beginning of the game when there is a lot of people and the stakes aren't too high and then, as the players really start to dwindle, maybe shift the times a bit.
I don't know though, I've enjoyed the game and had fun anyway we've done it. The timing isn't too big a deal.