I wish you could just put the stolen items into a horsebag but it would force the horse to walk speed so you can't just speed away.
That or once the stolen items are picked up, the timer starts ON THEM, not on you holding them in your inventory. That way you pick them up, droploot, reequip...