Following Gary's critique of his performance tonight, Rylan finally gave up with being polite to the Take That singer.
He told Dermot: "Forget about Gary, I'm not bothered anymore."
The eccentric singer had performed another energetic and eye opening routine for tonight's theme of 'Club Classics'.
[Related Story: Gary Barlow: Rylan Clark nearly made me quit the X Factor]
After performing a mash up of 'One the Floor', 'Don't Stop the Music' and 'I See You Baby', Gary still wasn't won over by the Essex singer.
He told Rylan: "I'll tell you my problem. I think we have ten very talented contestants and for every week you stay in, one will go."
Referring to this week's theme, Gary went on to say he was disappointed by Rylan's attempt.
He explained: "You said this was your thing, You should have been the best tonight, you were the worst."
Nicole Scherzinger stood up for her act, telling Gary: "It is his week- it's clubs classics and dance week- he's the only one getting up and dancing and singing at the same time."
Nicole then added: "You say he doesn't have X Factor. You're right because he has the X Y Z factor honey!"
Hmm, we're pretty sure Gary would agree on the Y factor part.
The X Factor results show returns Sunday night on ITV1.