Mine almost always will completely swell within 4 5 hrs. Maybe I've been really lucky with fresh yeast, though I doubt it. I normally smack the yeast pack at the start of the mash, and try and keep it in a warm place to help it on its way. I don't think you want to leave the pack to long, if you do the yeast may have eaten all the nutrients from the smack pack and decided to go back to sleep. You really want to pitch when they are warming up, getting ready to party and eat some fermentable, not when they have already gone to town, and are too hung over / tired to go for more. Remember just because the pack has swollen doesn't mean its good to pitch, particularly if you put it back in the fridge, that would almost definitely put them back to sleep, and it's only swollen because they were previously active. In saying that I agree that it would probably take longer than four hours for the yeast to peak in activity, but in my eyes, as long as the pack is bulged and they are happy yeasts looking for more, your good to go. Just my 2 cents anyway.
Its probably worth noting I dont make a starter. I dont think there is any point with the Wyeast smack pack, because its like a mini starter of its own, plus I havent had problems with attenuation.