Celebrity websites TMZ.com and Us Weekly quoted unnamed sources as saying Cole, 27, a member of the British pop band "Girls Aloud", was being replaced in the new Fox television venture created by Simon Cowell.
Fox said it had no information and representatives of Cole, Cowell and the show's producers did not return calls for comment.
The much-hyped "X Factor", created by former "American Idol" judge Cowell, is the highlight of Fox's fall TV schedule. It reunites Cowell with his old "Idol" sparring partner Paula Abdul and offers an unprecedented $5 million prize to the winner.
The judging panel was finalized just three weeks ago.
TMZ.com, quoting sources close to the production, said Cole was dropped because producers were worried that her strong northern English accent would be too difficult for American audiences to understand.
A source close to Cole told Us Weekly; "it's true. She isn't on the show anymore. Everyone is gobsmacked (shocked). This came out of nowhere. She had already been tested countless times. Simon had nothing to do with it. This is the network's doing."
Auditions before the judges began in Los Angeles on May 8 and have taken place in several other cities.