Last week, Kelly dropped the bomb of an allegation when she went on E!'s Daily Pop.
"I don't pinpoint [Meghan's] stuff out when she was having an affair with Jimmy when he was married to Allison," Kelly said. "I don't put that stuff out there. Sorry, spilling the tea now."
Prior to marrying Meghan in 2014, Jim was married to Allison Raski for six years. The two divorced in 2014, but Jim insists that he and his current wife were not sleeping together while he was still married to Allison, according to E! News.
"I think it's ridiculous first of all and second of all, we have a family and to sit there and say that (Meghan) was my mistress is beyond crazy," Jim explained. "This is the first time I've ever said anything to anyone but I just can't deal with the bulls--t lies anymore."
The sports broadcaster added, "I'm 100 percent sure she's fully aware that she's making up lies and the information she got was incorrect."
Jim said he was trying to "do the right thing" and protect his family by speaking out so strongly against Kelly's accusation.
"I just think that there should be repercussions for people who make up flat-out lies just to put themselves on TV because they have nothing better to say. I am tired of listening to Kelly's lies," Jim said.
Happy third anniversary to my partner for life. Even though we both forgot until this afternoon 🙈 I still think you’re the cherry on top of my Ted Drewes 🍒 and you look especially 🔥🔥🔥 in this pic (that you picked when I asked you to email me “a wedding photo”... 😂#husbands) Cheers to a million more years of forgetting our anniversary because time flies when you’re having fun, boo 😘 @jimedmonds15
The former MLB player continued, "I'm not looking for publicity, I just can't let my wife sit there and be killed and have to worry about this bullsh-t when she has a baby and all this IVF stuff going on."
Jim threw in a jab at Kelly as well, bringing up her recent divorce filing in his response.
"Her relationship is completely a disaster. She's filed for divorce and moved out and was having her own situations. I think she should worry about her own damn self," Jim explained to us. "If you're going to spread rumors about somebody, they better be true or you should have to pay some consequences for spreading lies."
Real Housewives of Orange County airs Monday nights at 9 p.m. EST on Bravo.