Mr Alexander welcomed the statement issued by Dave Hartnett following criticism of the senior civil servant, who had earlier rejected calls for an apology.
But Mr Alexander stopped short of saying Mr Hartnett had his "full confidence" when repeatedly pressed on the issue.
Instead he told reporters: "I have got full confidence in HMRC's ability to ensure that these issues are dealt with appropriately under his leadership."
Mr Hartnett, permanent secretary at HMRC, said on Saturday he was "deeply sorry" taxpayers were being asked to pay back an average 1,428.
But his comments were issued only after he caused fury by insisting the authorities were not to blame and declaring: "I'm not sure I see a need to apologise."
In a statement rushed out by the department, Mr Hartnett later said: "I am deeply sorry that people are facing an unexpected tax bill.
"Everyone in HMRC is working hard to make this as painless as possible. I apologise if my remarks came across as insensitive.
"I am working flat out with my colleagues to ensure everyone's tax is correct and the new computer system will help us do this.
"It was this new system that revealed the extent and size of reconciliations required and will help us be more accurate in future but we do not underestimate the distress caused to taxpayers and once again I apologise."
"He was right to apologise," Mr Alexander said. "This is a problem we inherited from the previous government and of course it's very unfortunate and I'm sorry that some people have got an unexpected tax bill."
An estimated 2.3 million people underpaid income tax in the past two tax years as a result of errors in their Pay As You Earn tax code.
The 900,000 owing the least were reprieved after the Government wrote off some of the debt but Ministers said the Government could not afford to waive the rest.