The first assumption SN made (as most do) is that I'm in love with the guy and want him to leave his wife to be with me.
Wow, I have a friend the same way and because it's a male/female dynamic that's the first thing most readers chime in about: that I have feelings for him or vice versa. I never call about him. They just pick up a recent male figure in my life who has been trying to contact me. When I ask if I know him previously, many start to describe him.
From my perspective I've never had any romantic feelings for him at all. We connected over a similar career and would talk about that at length and then we had a disagreement, fell out and I no longer want him to talk to him at all. Even outside female acquaintances thought we were involved that way, although I don't know specifically what I said or how they interpreted our conversations; nor what made them think that.
It seems to be an automatic response anytime opposite sex's are involved because I'm sure relationships feel the same way to readers whether it's a relationship with a co-worker (work-relationship); sister/brother (sibling-relationship); mother/father (parent relationship), etc.
In general, anytime I bring up another man (which I have about school or work) that is the first place the reader jumps to - intimacy.