Yeah certain its infection, the growth on the sides is there to stay. It has to be scrubbed very hard to get off.
I've kept good temps with all my batches, just under 20* for the ales, all batches tasted great after fermentation was complete. Its after bottling the infection seems to kick into...