Madison had a scar on his nose from frostbite.  He became afflicted with frostbite while out campaigning in 1788.  Side note: Madison was only 5ft 4inches tall.