In 1080, the Lords of Embrun were retainers to the local archbishops. Dispossessed of this duty and chased out of the town during a rebellion, they withdrew to their domains in the parish of St. Jean des Crottes, where they built a wooden keep at a place known as "Picomtal". The 1080 keep, a rustic structure made of larch beams, was only a look-out post surrounded by a stockade fence inside of which stood the scanty lodgings of the small garrison.

The building was part of a surveillance network over the main Durance road stretching from village to village all the way from Tallard to Briançon. By emitting signals from the upper levels of these watchtowers, the local population could relay information to one another in order to prepare for the arrival of any intruders, either by organizing a system of defense or by fleeing up into the high pastureland.

At the end of the 13th century, Hugues II and Boniface d'Embrun replaced the rickety, worm-eaten watchtower with the massive tower that stands at the southwest corner of the castle today. Like the previous building, it was a simple, uninhabited military structure where the lookouts stood guard after climbing to the top of a series of ladders.

In 1368, Louis, Duc D'Anjou and a Capetian prince, had a brush with Queen Jeanne of Naples, Countess of Provence. War raged in the Haute-Provence and Bas-Dauphiné for over a year. The Embrun region, which had been attached to the kingdom of France since 1349, was ruthlessly destroyed by the Provençal troops. It served as a lesson to them ...After this disaster, the region's villages and towns were seized by a frenzy of defensive construction to make up for their previous foolhardiness in totally neglecting to maintain their fortifications.

It was under such circumstances that Antoine d'Embrun transformed the solitary tower of Picomtal into a real military base of operations. Using the stone keep erected in 1270 as a starting point, between 1375 and 1380 he built a one-story rectangular castle delimited by four towers. The ground floor was composed of a kitchen and vaulted stable. On the first floor were two large-sized rooms which all the men of the garrison, as well as the farmers and their families, were packed into. The northwest tower was used as a staircase.

In 1417, Antoine de la Villette became the new lord of Picomtal. Like his predecessor, Martin de la Villette probably entrusted the expansion work to an Italian master builder between 1505 and 1510 for the sum of 5 000 livres. The two 125-year-old towers on the north side were razed, and what was left of the building was extended eleven meters to the north. The two 14th century towers were rebuilt identically on either side of the new façade. Finally, the entire structure was raised one story higher and the service staircase was brought inside the building.

Thus the castle has one tower that dates back to 1270, another to 1380, and two towers built in 1510, all very similar in look and diameter.