The Richmond MP told MPs on the Treasury Select Committee he was also concerned about the impact a rise in interest rates would have on the public finances, given the debt mountain built up during the pandemic.
He was being quizzed about his Budget speech in which he announced plans to extend the jov saving furlough scheme and other employment support to the end of September.
He also highlighted the potential for the economic "scarring" caused by the coronavirus crisis to be worse than the Office for Budget Responsibility's forecast for the economy to be three per cent smaller than it would have been in five years' time.
"So, even if the road map comes to pass and it turns out that the medium-term scarring impacts are worse than feared, that would be of concern because that is a very significant impact on the economy," he said.
"Obviously the exact path of the labour market is one that I'm constantly kept up at night by - sadly three-quarters of a million people have already lost their jobs and more are forecast to do so."
The forecast for unemployment to peak at 6.5% was lower than previously expected "but fundamentally that's a large number of people who are going to lose their jobs over the course of this pandemic and minimising the number of those unemployed and finding them new opportunities as quickly as possible is a thing that keeps me up at night".
He acknowledged the public finances "are much more sensitive to changes in interest rates and inflation than they were previously" and an increase in interest rates would have a "significant impact".
Treasury Select Committee chairman Mel Stride challenged Mr Sunak over the assumptions on public spending contained in the Budget, claiming they were unlikely to be met.
Mr Stride said that if he was chancellor he would be "very worried about how realistic the spending figures are", with "no explicit provision" for ongoing coronavirus spending, pressure on NHS finances, education, the rail network, the potential further extension of the £20-a-week Universal Credit increase and social care.
He asked the Chancellor: "Is it just not the case that the current numbers you have got in there on spending are unrealistic and are just going to come under unbearable pressure over the coming spending round and the years ahead?"
Mr Sunak said there had been a "significant uplift" in public spending and by the end of the parliament "the state will be spending historically very large amounts of money".
But the Office for Budget Responsibility has pointed out the Government has so far cut more than £15 billion a year from day-to-day departmental spending from 2022-23 onwards compared with earlier forecasts.
The Chancellor said "the size of the state, that's quite chunky by historic standards" but he acknowledged that choices on funding priorities would have to be made at the spending review.
"All governments have to prioritise within a pie, the good news is that the pie is growing at a rapid rate over the course of this parliament and relative to the size of our economy is a large pie," he said.
During the session Mr Sunak defended the hike in corporation tax from 19 per cent to 25 per cent in April 2023.
He told the Treasury Select Committee that there were other policies which would attract businesses to the country
"What I would say is corporation tax is not the whole shop window," he told MPs.
"Corporation tax is just one of the things in the shop window and there are lots of other things in the shop window as well that are going to entice people to come through the door.
"Even with the higher rate, it's still - on the OECD way that they look at them - the lowest rate in the G7, the fifth lowest rate in the G20, with only Indonesia, Saudi Arabia, Russia and Turkey below us."