If you want the chlorine/chloramine removed then it is necessary to use them, or the powdered form which I prefer. Used at the correct dosage I don't see a downside, the SO2 will just boil off when the wort is boiled, and you won't be risking those flavours.
Neutral or soft water like...