A last-gasp equaliser and a missed penalty cost Team GB a place in the semi-finals after Ellen White’s double had give them a 2-1 lead going into the final minute of normal time.
But a defensive lapse at the death allowed Sam Kerr to haul Australia level and take the match to extra-time.
Caroline Weir then missed from the spot before goals from Mary Fowler and Kerr sent the Brits spinning out, with White completing her hat-trick in vain.
“We are devastated right now. It is hard to lose a game like this,” said head coach Hege Riise
“We were well prepared. I think we played most of the game quite well. We created a lot of chances, probably should have finished a few of them. Now it’s quite hard. We are all devastated.”
She added: “We win and we lose together. So there is no-one to blame. We stood up and we fought together and it wasn’t enough from any of us. So we are disappointed that we couldn’t get the fourth goal and take it to penalties.”
Defender Leah Williamson reflected on a missed opportunity, saying: “It’s hard because this team isn’t a permanent thing. We come together once every Olympics if we’re lucky. Everyone takes so much pride in that.
“I’ll take so much pride forever that I was able to put on that shirt and play for this team as I’m sure the others are too. You have to take all of that. But we didn’t just want to come here and celebrate being Olympians. We wanted to win.”
Australia took the lead against the run of play after 35 minutes through Alanna Kennedy. White levelled after 57 minutes and Britain went ahead when White reacted quickest in the area.
In the 89th minute, Team GB hesitated fatally when the ball dropped to Kerr to equalise.
Nine minutes into extra-time, Nikita Parris fell under a challenge from her former Lyon team-mate Ellie Carpenter but Weir’s weak effort was saved.
Austeralia went 4-2 ahead inside the first minute of the second half, White pulled one back with another header for her sixth goal of the Games with five minutes remaining but it was not enough.