Blue said his firm was hired by Knight Takes King Productions to handle on-set security during House of Cards' first season in 2012 and 2013.
After hearing Spacey's comment, Blue told The Daily Mail he raised concerns with "The Powers That Be." He was told, "That's just the way he is; we've got to keep him happy."
Producers later told VIP Protective Services that they were happy with their performance, but their $1.1 million contract was not renewed.
Blue told the Daily Mail he plans to sue Spacey, claiming that he lost millions of dollars in potential revenue after VIP's contract was not renewed.
One of his employees, Eric Lyles, corroborated Blue's statements, adding that Spacey would not shake his hand or even acknowledge him.
"I was a tremendous fan of the man but now I am disgusted by him," Blue, who lives in Annapolis, Maryland, told the Daily Mail. "I lost faith that I am in the right job after the contract was terminated. I questioned myself for months. What could I have done differently? Then I came to the realization that I did everything I was supposed to do, I was just dealing with a racist man."
Blue said he decided to finally tell his story because "when it comes to the point where people are being harmed, enough is enough."
Blue claimed his relationship with Spacey got off to a rocky start when he and Lyles were introduced to the Oscar-winner. He claimed Spacey never spoke to his security guards, but would talk to everyone else, including police officers.
"I always meet the stars and see what they're like, what their likes and dislikes are," Lyles explained. "When I went to meet Mr Spacey to say hello to him, I went to extend my hand to him to shake his and he didn't want to shake my hand."
Blue claims Spacey always locked the door to his trailer whenever he left to go on set. He said Spacey never even replied to him when he said "Good morning," or "Good afternoon."
He would look at me, walk back to his trailer and check to make sure the door was locked.
"Over the course of that day I started laughing at him. He did it every single time he left his trailer - at least a dozen times," Blue recalled.
Blue said that after the contract ended in April 2013, VIP got rave reviews from the production. However, Blue said there was a different meeting where Spacey said "he did not want n— on his set anymore."
Blue was not at the meeting, but was told about this by a production manager who was. After the meeting, the VIP contract wasn't renewed.
"There were no break-ins. I pride myself on our service and integrity so to hear these lies has struck a cord. No reason was given for terminating the contract," Blue said.
Netflix, Spacey's representatives and Media Rights Capital, which produces House of Cards did not comment.