Bruce Clegg, 49, threw himself into the DIY project to cope with the death of son Nathaniel Hemingway-Clegg, 20, from type one diabetes complications.
The 16-square-metre wooden structure cost around £1,750 to build and sits in the back garden of his semi-detached council-owned house.
It is kitted out with a bar named after Nathaniel and features family photos and his favourite Harry Potter memorabilia.
But a neighbour complained about "antisocial behaviour" prompting a site visit from Wakefield District Housing, and he was ordered to tear it down, piece by piece.
Workers said it was "unsafe" and too close to the neighbour's fence, despite warehouse worker Bruce pleading it was vital to his grief recovery.
Father-of-three Bruce said he's preparing to rip it apart - but fears it will leave him without anywhere to safely and happily remember his son.
Bruce, from Wakefield, said: "Losing a child is the worst thing that can ever happen to a person - Nathaniel was a great guy with a good heart.
"That's why I wanted to build the summerhouse in his memory - it is a place to go to have a sit down with a drink, look at some photographs, and remember him.
"When any of us go in there, it makes us feel like he's still around. I began building the summerhouse to deal with the grief of losing Nathaniel and it never even occurred to me we'd need permissions because I was just thinking of my son.
"Now it seems we're going to have to start again from scratch, or lose the thing which has brought my family so much comfort altogether. We need the summerhouse to help us through this tough time - it has helped me to stay strong for my kids."
Bruce began building the memorial summerhouse in June of 2020, a month after losing his eldest son.
Despite having been diagnosed with diabetes aged just one, Bruce said Nathaniel, a trainee chef, sometimes struggled to control the condition as he grew older.
As a teenager he suffered diabetic ketoacidosis, where harmful substances called ketones build up in the body as a result of a lack of insulin.
But due to fears of catching Covid-19 in hospital, Nathaniel kept his ill-health a secret, and eventually died at home on May 2 last year.
Bruce, who is also father to Craig, 19, and Michael, 17, threw himself into the DIY project as a way of processing his grief as well as honouring Nathaniel's legacy.
The family spends time in there whenever they were missing their brother as it would make them feel "like Nathaniel is still around."
But in April this year - just as Bruce was adding the final touches to the house - a complaint from a neighbour prompted Wakefield District Housing (WDH) to visit.
Representatives said he hadn't asked for permission, and had to tear it down until he had it.
Bruce said: "In my grief, I had never even thought about permissions as I just wanted to do something for Nathaniel. Looking at the tenancy agreement never even crossed my mind.
"I've tried so hard to change their minds, explaining how much the summerhouse means to my sons and I, but it hasn't worked. Going in there makes me feel teary and it's the perfect way of remembering Nathaniel and we all just want to let his memory live on.
"We have lost a huge part of the family so this summerhouse means so much, it's so much more than wood to us all. It kept us all on an even keel as we dealt with Nathaniel's death - and having that project was the only thing that kept me focused and strong enough to support my other sons."
Mick Walsh, executive director of housing for WDH said: “Firstly, I’d like to share my deepest condolences with Mr Clegg and his family following the passing of his son. During what I’m sure has been an incredibly difficult time, I’m pleased that our Wellbeing team were able to visit Mr Clegg and his family, to offer support.
“It’s very important to us that our customers are able to feel like they can make a house a home when living in our properties, which includes being able to make home improvements.
"When considering home improvements, the safety of our customers, their neighbours and anyone who visits their home is of utmost importance, which is why we ask all our customers to seek our permission before they start work, as part of their tenancy agreement. This helps to make sure any work is carried out to a high standard, which is why we challenged this particular structure.
“In this case, Mr Clegg had not sought our permission beforehand. Following a complaint of antisocial behaviour, our surveyors and estate managers visited the property and assessed the structure as unsafe due to the quality of; materials used, electric works installed and proximity to the boundary fencing.
"We have asked Mr Clegg to take down the structure, which he has agreed to do. We have offered to work with him to provide guidance on the materials to use and regulations to follow so he can re-build his memorial to his son, in the safest way possible.
"This is of course, subject to Mr Clegg taking down the original structure, submitting a home improvement permission request, and addressing the initial antisocial behaviour issues reported.”