"I said to Kelsea, 'We should do something completely different,'" Urban told the crowd ahead of their performance.
"It's a curveball," Ballerini chimed in.
"I'm a big fan of this song," Urban continued, "and I said to Kelsea, 'We should try this song — what the heck.' It's a great song, so we're gonna give it a shot."
The duo then launched into Camila Cabello's latest single, "Never Be the Same," a looping pop track that saw the pair share the lead vocal role and team up for a few harmonies.
Cabello later posted about the pair's performance on her Instagram, and the star was clearly excited about the cover.
"OH MY GOD!!!!" she wrote. "@keithurban AND @kelseaballerini THIS. IS. MAGICAL."
Urban is bringing Ballerini along on the North American dates of his Graffiti U World Tour 2018, which he announced during the surprise show at Nashville's Exit/In. The tour will support the release of his upcoming album, Graffiti U.
Ballerini told the crowd that she was inspired to pursue country music after hearing Urban's song "Stupid Boy" when she was younger.
"I was starting to write songs and I didn't really know where they fit," she said. "And then I went to my friend's MySpace page and I heard the song 'Stupid Boy,' and it changed my life and it moved me to Nashville and I really credit so much of my path from hearing that song. So being able to tour with Keith Urban is one of the craziest full circle moments of my life."
The singer echoed that statement in an Instagram post she shared after the show.
"I wish I could go back and tell my 14 year old self that 10 years later I would spend my summer touring with the person who's songs led me to Nashville," she wrote next to a photo of her younger self meeting Urban. "YALL. see you on the @keithurban tour!!!"