One probation officer was even propositioned by a white male colleague because "he had not had sex with a black woman before", a report on race equality in the service found.
Chief Inspector of Probation Justin Russell said the findings were "disappointing" and said staff did not feel experiences of racism had been taken seriously.
Inspectors said they heard "distressing stories" of inappropriate behaviour towards ethnic minority staff including instances of "stereotyping, racist and sexualised language, and false allegations".
Several said they did not feel it was safe to raise issues of racial discrimination and serious complaints had been "repeatedly downplayed, ignored or dismissed".
Although inspectors described some of the problems as "systemic", Mr Russell stopped short of branding the Probation Service "institutionally racist".
The report found a "reluctance at times for managers to call out racism for what it is", with inspectors hearing of complaints of "racist language being found to be swearing, or racial slurs dismissed as banter", adding: "The staff affected were distressed that their experiences of racism had not been taken seriously."
The woman who was propositioned told inspectors it was an example of her experience of "oppression, alienation, exclusion, isolation, bullying and harassment".
The report added: "Although some black, Asian and minority ethnic staff (BAME) may be willing to work with such cases, managers cannot presume this, and the potential for reliving past trauma or experiencing further racism must be recognised."
More than 222,000 offenders are supervised by probation services across England and Wales and around a fifth are from BAME backgrounds.
The inspection found staff lacked confidence in addressing issues of race with BAME offenders.
Mr Russell said: "Some officers, by their own admission, avoided talking about these issues altogether."
He added that there were concerns about "every stage of probation supervision".
He said: "Probation officers need to find out as much as possible about individuals to support their rehabilitation."
Probation's focus on racial equality has "declined", the report said, since the service was part-privatised by then justice secretary Chris Grayling in 2014.
Mr Russell said the change would be an "important opportunity to reset and raise the standard of work" with BAME offenders and staff.
Napo, the trade union for probation staff, said it was "deeply shocked" by the findings.
General Secretary Ian Lawrence said: "I find it deeply disturbing that our members are being discriminated and disadvantaged in the workplace."
He added: "There is a lot of ground that needs to be made up once the service is reunified in June to tackle disproportionality for BAME people in the criminal justice system."
The inspection, carried out between October and December, looked at services in Bradford and Calderdale, Liverpool and Sefton, Hackney and Tower Hamlets in east London, Bedfordshire and Birmingham, considering 150 cases while also speaking to more than 100 probation staff and 80 offenders.
Inspectors made 15 recommendations for the Prison and Probation Service and the National Probation Service, which included calls for more training and a plan of how to work with BAME offenders.
Director general for probation Amy Rees said it was a "difficult" report to read and support "clearly" needs to be "better tailored" for BAME offenders.
She added: "We are working hard to diversify our workforce so that we have greater collective understanding for the particular challenges faced by ethnic minority offenders, and I want to reassure probation staff that we are listening and acting on their concerns."