David Cameron has been forced to defend Chief Whip Andrew Mitchell again amid fresh demands for him to be sacked over his confrontation with police in Downing Street.

Labour leader Ed Miliband used the first Prime Minister's Questions since the incident last month to denounce Mr Mitchell, saying he should have been arrested like any other "yob" who had sworn at officers.

And at Wednesday night's weekly meeting of the Conservative backbench 1922 Committee at Westminster, four Tory MPs were said to have voiced concerns about Mr Mitchell's position.

James Duddridge, Sarah Wollaston, Anne Main and Andrew Percy were reported to have been outnumbered by up to 15 who spoke out in support, led by Tory grandees Sir Peter Tapsell and Bernard Jenkin.

Even some of those said they had concerns but took the view that it would be "a spectacular own goal" if Mr Mitchell was forced to go now. Senior sources have acknowledged that the whole affair had been damaging for him.

"The sense is he will need to rebuild relationships," one source said. "You don't come through something like this emboldened."

Overall, however, the mood was said to be that it had "gone on long enough" and it was time to draw a line under the matter. Mr Duddridge, a former whip who was sacked in last month's reshuffle, was said to have called for Mr Mitchell to "do the decent thing" and "fall on his sword".

Earlier, in the Commons chamber, Mr Miliband went on the attack, claiming Mr Mitchell was "toast" having clearly lost the support of senior colleagues.

"While it is a night in the cell for the yobs, it is a night at the Carlton Club for the Chief Whip. Isn't that the clearest case there could be of total double standards?" he said. "His position is untenable. In other words, he is toast."

But Mr Cameron insisted that Mr Mitchell had apologised to the officer concerned and that the apology had been accepted. "What the Chief Whip did and what the Chief Whip said were wrong. I am absolutely clear about that. That is why it was important that the Chief Whip apologised and that apology was accepted," he said.