Celtic raided Brighton to snap up Duffy on a season-long loan deal during the last transfer window and the Republic of Ireland international was tipped to add solidity to the backline at Celtic Park.
Neil Lennon’s men have struggled defensively though and are now 19 points behind league leaders Rangers, although they boast three games in hand.
Rae understands the reasons behind Celtic swooping for Duffy, but believes the loan has not worked and the Bhoys should ship him back to Brighton.
“When I look at the players at the start of the season, [Christopher] Jullien and [Kristoffer] Ajer and people are saying we need a real out and out centre-half, somebody who goes and heads the ball because these two guys don’t do it”, Rae said on Clyde 1’s Superscoreboard.
“I feel like the whole lot of them [at the back] have dipped. Jullien has come off the pace when he’s been fit, Ajer’s off the pace.
“Duffy hasn’t hacked it and if it was me I’d send him back in January.”
The defender started on the bench during Celtic’s 1-0 loss at Rangers on Saturday, bring drafted into the action in the 65th minute.
Celtic forked out a loan fee of £2m to sign Duffy and it is unclear if the loan deal contains any option for the Bhoys to terminate the deal early.