Departing prison governor 'hard act to follow' - minister