You'd have to have a save game before you gave the keys to him, otherwise I don't think there is a way in.