The Morley and Outwood MP, one of the former Prime Minister's closest allies who is now vying to succeed him as Labour leader, said: "We were making a mistake by brushing it under the carpet."
The disclosure came as the Shadow Education Secretary admitted that immigration under Labour had affected the pay and conditions of "too many".
He insisted Labour had to acknowledge its mistakes and called for better protection for UK workers under any future expansion of the European Union.
He went further than other ministers from the previous government in acknowledging public anger about immigration from Eastern Europe.
The issue is thought to have cost Labour many votes among its traditional working class supporters.
Mr Balls said on television that it was something he had raised directly with Mr Brown.
Gillian Duffy, the Rochdale pensioner branded a "bigot" during the election by the then Prime Minister, had only voiced the opinions of many Labour supporters, he added.
"I said to him that you should be talking about immigration, the last year and a half, and that we were making a mistake by brushing it under the carpet," Mr Balls said.
"I think Gordon's answer to Mrs Duffy showed he'd not been having the conversation, because what she said was the kind of things being said by Labour supporters, and in some cases former Labour supporters, over the last year and a half which was 'look, we're not racist, and we support our EU membership and we know that immigration's important'."