This has confused the hell out of me too but my understanding is that you base it on the highest temperature the brew reached after primary fermentation has finished. If you did a diacetyl rest at 18-20 for example then you would prime as per that. If it sat at 12 and was then crash chilled you would prime for 12. Very short term fluctuations should be ignored (eg if the brew got up to 16 for 30 mins).