Personally I wouldn't touch it and leave the glad wrap where it is, as long as it has not come out under the rubber band yet. Nothing should've come in, just another possible risk of infection if you take it off. The yeast gunk around the neck is no different to the yeast gung inside the fermenter once high Krausen of a "normal" ferment subsides, and you wouldn't clean that off either.
I would just spray the yeast off the sides of the fermenter with plenty of no rinse sanitiser. If however the glad wrap is not completely held by the rubber band anymore, follow Felten's advise, but do it as quick as possible.
Florian