Simon Cowell is accused today of blaming Gary Barlow for The X Factor's drop in ratings this year.
The show, which pulled in just under 20 million viewers for its 2010 final, has been appealing to around 8-9 million per episode this year.
Speaking to The Times, a source said: "Cowell doesn't think that [Barlow] knows enough about who to choose from the auditions.
"The one thing he did after the 2010 show was to create One Direction, who are now the biggest pop group in the world."
It is also thought that the mogul has "another man in mind" to replace the Take That frontman, after being unwilling to even sign him up for a second series this year.
Still, Barlow has remained a popular judge on the show throughout this series, competing with Nicole Scherzinger for the title of Best Judge in a number of online polls.
His last remaining Over 28s act, Christopher Maloney, is competing for victory this weekend.
Was Christopher Maloney's X Factor Finals Failure Due To Drag Queen Karaoke Session?
Harry Styles Leaves One Direction Bandmates To Face Airport Chaos As He Flies Into UK On Taylor Swif
Rihanna 'Furious After Chris Brown Parties With Ex Karrueche Tran'