Must I correct myself to the person I was speaking with if I uttered a statement of disbelief in front of him—such as swearing by other than Allah unintentionally—or is it sufficient to retract it in my heart?

Whoever swears by other than Allah should say, "There is no god but Allah," which is an expiation for that. This utterance is obligatory along with repentance and remorse for what was said, and a firm resolve not to return to it, and not to glorify anyone other than Allah. Swearing by a created being is a minor shirk that is forbidden, unless it was unintentional, in which case there is no sin upon the one who did it.