Robert Koren (rib) missed out but Tom Cairney (foot) played after shaking off a minor injury.
West Ham, who needed to win and hope Southampton lose at home to Coventry, took the lead on 36 minutes through Carlton Cole.
But with the Saints winning 2-0, the Hammers are set for the play-offs.
Four minutes after half-time West Ham doubled their advantage and it was the former England striker Cole on target again.
The Tigers pulled a goal back nine minutes from time through Corry Evans, but it was too little, too late.