There are other people you hang around, and I can almost bet you that they don't like me or even know who I am. And if I get to know them, they probably regret it.

[He's not trying to make Zuko like him any less, but just point out how he's lacking in the ability to make friends. He has no idea why Zuko likes him, but the guy is stubborn.]

I can't tell you that. [His reply is quick and firm, voice gentle in that he doesn't want Zuko to ask any more than that.]