'Hasett's Fort' was an early 17th century castle and bawn built by Francis Blennerhasett of Norfolk as part of the Ulster Plantation.  It was renamed Castle Caldwell after James Caldwell purchased it in 1671.

James, who became Baronet in 1683, was a wealthy Enniskillen merchant and trader, whose family came from Ayrshire. One of his descendants, another Sir James Caldwell, spent at least £16,000 in c.1778 building or rebuilding the house and transforming the surrounding estate, by the shores of Lower Lough Erne, into elaborately landscaped grounds.

Sir James Caldwell was also responsible for erecting a fiddle-shaped stone in memory of the fiddler, Denis McCabe.

In 1792 his son, John, renovated the castle in Gothic style, rebuilding the south side which had become ruinous, incorporating the two western flankers of the original 17th century bawn and adding a museum building on the east side.

In 1830, the Estate passed through marriage to the Bloomfield family.  In 1857, John Caldwell Bloomfield co-founded the Belleek Pottery.

The image is a photograph taken around 1900 by Canon WH Dundas of a group in front of the Castle ruins.