From the older Iron age and onwards traces of humans can be found around Kagghamra. The two most impressive remnants are the two ancient castles (hillforts) that are situated on two small mountains close to Kagghamra. Their names are Korpberget and Strömskullen. Both have an amazing view over the sea. Korpberget (Raven Mountain) still has walls that is easy to see and also a grave on the top of the mountain. The grave might be even older than the older Iron Age but it is impossible to say something for sure. An old farmstead, Brubbhamra, is situated close to Kagghamra. It has its roots in the younger Iron Age and several gravefields are situated nearby. The place was deserted in the late 18th century but still can traces of the houses that stood there be seen as terraces and small line of stones. Kagghamra starts to become known as a manor around 1650 AD. It gets the formal rights to call itself a manor 1680 AD. In this time Kagghamra was called Sjöberg. The building that stands there today was built in the first half of the 18th century. The probable year of construction is 1731 AD. The building is white with a hipped roof and three chimneys. The building style is often called rokoko. Today it is privately owned. It was at Kagghamra that the first industrial initiative took place. In 1875 a certain Gustav Cavallin bought the place to start a ship construction industry. The houses, except the Manor house, that stands there today were originally intended for workers at the place.