However, now it seems that Simon was telling porkies in order to save face and the reputation of the show.
Talking to TMZ about the incident, Simon confessed that Britney had in fact had a bit of a wobble during her first day.
He said: "No one is going to admit that on the first day your highest paid star after twenty minutes has walked off set."
"[She] had a moment on the first day where she said to me 'I'm not sure I can do this'."
Luckily, Simon managed to use his powers of persuasion to talk her round- and considering he had $15 million riding on Brit, it's a good thing he succeeded.
During the time of the incident, Britney was also quick to refute the allegations.
She tweeted: "'#Britneywalksoff??? LOL was just taking a little break people. I am having the BEST time!!!"
[Related article: Britney Spears dons a bikini]
[Related article: Britney Spears gets her flirt on with LA Reid]
However, Britney now seems to have got to grips with her role as a judge- with Mr Nasty himself claiming that she is proving to be a formidable judge.
During a press conference this week Simon admitted: "She's surprisingly quite mean"
"She can't say no quick enough. She's quite difficult to please."
Reality star Khloe Kardashian is currently the front runner to take over hosting duties from Steve Jones after he was relieved from his job at the end of the last series.