A couple of points.
If your starter is made from poor quality malt or fermented at the wrong temperature, this will lead to off flavours in your beer if the whole lot is pitched. Some of my earlier ag brews had extract flavour in them from the starter.
If you pitch the whole starter, try and pitch it at full krausen, that point where the starter is shaken and you see lots of activity for the first time.
If you brew up a starter, let it ferment out and just pitch the slurry, this will let you brew as big a starter as possible, ferment at higher temperatures and aerate a lot.
Have used both methods, and my preferred method is to let it ferment out, or as outlined a few days ago in the "cold ferment" thread.
Your choice will depend on what level of brewing you are at, how much equipment you can devote to starters and your brewing time frame.