Fair question, the fears are well- founded. I'll soak just about everything in perc (i.e. Napisan, sodium percarbonate) every few months, but it is almost impossible to rinse all of the residue off with our tap water here as it is neutral to alkaline, so I now add 1 tsp citric acid to a bucket of warm rinse water. Works brilliantly and uses a lot less water overall (so beneficial where water is scarce), just do a final rinse with plain water then drain (and sanitise later).
Once I did a batch of bottles with this perc soak and rinsed with just town water and not very thoroughly, later I started sanitising them with Starsan and part way through the batch the foam had vanished and it had gone cloudy, I guessed it was being neutralised by the perc residue so I had to rinse them again and then re- sanitise, that's when I started using the citric acid rinse. I'd imagine any weakly acidic solution would do, sometimes I use citric acid to adjust pH of the mash and sparge water so I have it on hand.
Oh, and plus what TB says!
Edit: Rinsing effectiveness depends upon your rinse water composition, most folks will use reticulated/ town water, here it is not as effective as it could be so I give it a helping hand with citric acid, but it may not need that in other locales.