You need some form of speed control over the fan. At full bore (at least on mine) the neo magnets often throw the stir bar, so I dial it back until it doesn't rattle about. The effect is more pronounced with less liquid in the flask.
You could get away with a variable DC power supply (jaycar sells one that goes from something like 3-15V in about 5 steps), or add some resistors in, but you're best bet is just getting a PWM.
There only about $3 - $5 from eBay. This is one I bought. Or get one of the DIY kits from digitalhomebrew.com. If was to do it again, I'd buy the complete kit ($29 - just needs a housing & control knob) but you can also just buy the controller separately for $15.