It's not the time that's important Brewkid, it's the yeast 'condition' for want of a better term. You can either make the starter with the intent to ferment it out or to pitch at high krausen. The speed at which this happens will depend on yeast health (mainly age), temp and wort conditions.
If you have a stir plate and intend to ferment out the beer is being oxidised with the intent of increasing yeast count. This process typically takes 24-48h with fresh yeast at ~25°C. Once fermentation is finished, it should be allowed to settle as described above so you are not tipping out the babies with the bath water.
If you pitch at high krausen, you are in effect getting the yeast up and running and presenting them to fresh wort before they have finished their growth cycle. It's important to consider what your starter is made from in this case as it will be in the final beer. Depending on who you speak to, there are benefits with both approaches.
In your case I would give it 72h. Give it a shake, and if it bubbles and foams it's still finishing. Wait until finished, allow to settle in the fridge, decant and enjoy.
PS: I use a small PVC hose to siphon the spent wort off rather than tip the flask. Sucks up almost every last drop and preserves almost the entire cake.