At a judicial review of the intelligence service’s activities, lawyers for human rights group Liberty said that MI5 had acknowledged internally that data was being mishandled.

The Investigatory Powers Act (IPA) – known as the Snoopers’ Charter – provides the security services with powers, under warrants issued by judicial commissioners, to hack computers and phones and intercept people’s communications.

These powers allow the government to carry out “bulk surveillance” on people who are of no intelligence interest. That information is then stored by the security services for potential investigations in the future.

The Investigatory Powers Commissioner’s Office (IPCO) is responsible for ensuring that privacy protections contained in the IPA are upheld, including that safeguards around storage and timely deletion of data are met.

Following the initial revelation last month that MI5 had breached IPA privacy safeguards, a series of 10 documents and letters from MI5 and IPCO – disclosed during the course of Liberty’s ongoing legal challenge to the IPA – have revealed more detail of those breaches, including that MI5 has failed to meet its legal duties for as long as the IPA has been law.

Despite heavy redaction by MI5, the documents reveal how a number of failures have led to what the Investigatory Powers Commissioner, Lord Justice Fulford, has concluded is the “undoubtedly unlawful” conduct of the UK’s domestic security service.

The commissioner concluded that MI5’s method of holding and handling people’s data was “undoubtedly unlawful”, saying: “Without seeking to be emotive, I consider that MI5’s use of warranted data… is currently, in effect, in ‘special measures’ and the historical lack of compliance… is of such gravity that IPCO will need to be satisfied to a greater degree than usual that it is ‘fit for purpose’”.

The UK government has applied for further details on MI5’s breaches to be provided to the court through secret evidence and private hearings.

Megan Goulding, Liberty lawyer, said: “These shocking revelations expose how MI5 has been illegally mishandling our data for years, storing it when they have no legal basis to do so. This could include our most deeply sensitive information – our calls and messages, our location data, our web browsing history.

“It is unacceptable that the public is only learning now about these serious breaches after the government has been forced into revealing them in the course of Liberty’s legal challenge. In addition to showing a flagrant disregard for our rights, MI5 has attempted to hide its mistakes by providing misinformation to the Investigatory Powers Commissioner, who oversees the government’s surveillance regime.

“And, despite a light being shone on this deplorable violation of our rights, the government is still trying to keep us in the dark over further examples of MI5 seriously breaching the law.”