At the library I was sitting next to a gentleman who I guess had put on some kind of talc after shaving. It was a pleasant smell, but it's the kind of thing you have to pretend not to notice. You can't just tell another man you don't know, "Hey man, you smell good." It could lead to misunderstandings, and ugly situations in general.
Of course if you compliment a female stranger on her perfume she might think you're creepy too. I thnk you probably take a slightly different path to discomfort there.