In this new sneak peek from this week's I Am Cait released by E! News, Scott finally speaks out during a candid conversation with the 65-year-old.
"What do you think...from a guy's perspective? Because everybody seems to be very supportive - and this and that - I don't know it's just a guy thing, I've just struggled with it," Caitlyn explains to Scott.
"I definitely think it's a lot different from me than a buddy you know because the sense that you've been in my life a long time. You're part of my kids' life," Scott explains. "I mean to be completely honest...I don't know if every man in the world is going to be as comfortable as I am."
Caitlyn uses Kris Humphries' untimely Twitter comments as an example of how some people haven't been as accepting, although after that incident Kris claimed he wasn't talking about Caitlyn - the damage had still been done.
"Anybody who is worth being in your life will understand and will work at the relationship," Scott reveals and Cait agrees, saying she has so much good going on in her life right now, she doesn't have time for haters.