Daniel Jones, 29, is accused of starting the blaze in which Andrew Broadhead and eight-year-old Kiera died in an attempt to destroy evidence of a burglary at a house across the road.
But Jones has told a jury at Leeds Crown Court that he did not light the fire at the home in Wakefield, West Yorkshire, or carry out a raid on the other house.
The court has heard that Jones was worried CCTV footage from the house in Ash Crescent, Stanley, would reveal he had stolen a safe from a friend's home on the same road a few days earlier.
It is alleged that he poured petrol through the front door letterbox and set it alight in the early hours of October 19 last year, when Mr Broadhead, his wife Sara, and their daughters Mia and Kiera, were asleep.
Mrs Broadhead and 12-year-old Mia managed to escape but Mr Broadhead, 42, and Kiera were overcome by the rapidly spreading smoke and fire and pronounced dead at the scene.
The trial heard that a tracker on Jones's van showed he had parked the vehicle on Aberford Road, a short walk from Ash Crescent, between 3.28am and 4.22am on October 19.
But he initially told police he had been asleep at his father's house in Knottingley.
When asked about his answers to the police in court, he agreed he had been lying.
Giving evidence wearing a pale grey suit, Jones told the jury: "I panicked with the interviews. I'd never been in that situation before, the night before I'd been out all night and not slept. I'm sorry, I made a mistake, I just panicked at the time."
Jamie Hill QC, defending, asked him: "Were you lying because you are guilty of any of those offences?"
Jones answered: "No, not at all."
The court has seen CCTV footage of Jones buying 3.75 litres of petrol at a garage in Leeds shortly before travelling to Stanley.
He told the jury he bought the petrol for a friend's motorbike and was going to help him move the bike and some cannabis plants from a field.
He said he waited for the friend but, when he did not arrive, he went home.
But in cross-examination, Jonathan Sharp, prosecuting, told the court that Jones's friend said he did not own a motorbike at the time.
He described the defendant's reasons for buying petrol and being in Stanley in the early hours of October 19 as "nonsense" and said he had made up the story after hearing the evidence against him.
Mr Sharp said: "You waited for the Crown to produce all its evidence. It's then, and only then, that you started devising a story that will fit it."
Jones told the court it was a coincidence that he left the area just a few minutes after emergency services were called at 4.20am and denied that CCTV footage of a person seen in Ash Crescent that morning was him.
He told the jury that he carried out a number of Google searches about the fire the following day because it was news from his local area.
Jones, who admitted regularly smoking cannabis and occasionally using ketamine and cocaine, said he did not have a drug problem but had been to Gamblers Anonymous meetings and was planning to declare himself bankrupt because of the amount of debt he was in.
But he told the jury that he did not steal the safe, containing £850 and 1oz of ketamine, from his friend's home.
Jones, of Spawd Bone Lane, Knottingley, denies two counts of murder, alternative charges of manslaughter and burglary.
The trial continues.