The incident happened at about 4pm yesterday on Ingham Road, in the Thornhill Lees area of the town.
West Yorkshire Police today said the girl was with her 16-year-old sister when the man approached them and “took hold” of her.
He also assaulted the 16-year-old before letting the younger girl go and making off in the direction of Parker Road.
The man is described as white, aged around 40 and balding with blond hair.
He was wearing a cream-coloured hooded-style top, light coloured trousers and black shoes or trainers.
Kirklees CID’s Det Insp Andrew Lockwood said: “We are treating this incident extremely seriously and are carrying out extensive enquiries to trace those involved.
“It is unclear at this time what the man’s intentions were but this incident was extremely distressing for the two girls involved and will obviously cause concern in the local community.
“Our neighbourhood policing team have increased their patrols and there will be a visible policing presence in the area.”
Anyone with information is asked to contact West Yorkshire Police on 101, quoting crime reference 13180566405.
Information can also be passed anonymously to independent charity Crimestoppers on 0800 555111.