After the fire of 1890, the small structure was knocked down, and all larger stones used in the foundations of the rebuilt Keepers' House. Legend has it that the men who worked on the rebuilding broke the Boathouse into pieces during an enforced stay at the Point during bad weather. Mystery clings to the claims in this story like sea fog. Locals despaired at the use of "ill fated stones" in the making of a new building. As an added twist to this tale, it is said that all three of those workers were "stark raving mad" by the time the winds eased.