I moved here from Sydney over 12 months ago and am not going back anytime soon. Every city has its issues be it Fortitude Valley or South West Sydney. Its frequently about perception and you have to be street smart wherever you are and not be part of the problem.
I got hit on mutiple times every day walking from Elizabeth St through Central Railway to work. In India it hardly happened at all ( I think maybe twice). I found Central Railway infinitely more threatening than any of the gaols I've worked in. Our friends were horrified when we went to the middle east after 911. The day after the US started bombing Iraq, R and I travelled through the Palestinian refugee camps east of Amman, treated with complete respect.
Try driving up the coast from Cairns to Port Douglas and not be impressed