Understanding LLMs and their data protection risks: A Working Paper by the “Berlin Group”

On 6 December 2024, the “Berlin Group” published a Working Paper on Large Language Models (LLMs). The aim of the paper is to provide guidance on how to address data protection risks posed by LLMs, emphasizing that that this can only be achieved by having an effective understanding of how the underlying technology works.  The paper provides valuable insights into LLM risks for Data Protection Authorities (DPAs) and organisations developing and deploying LLM technology. It makes clear that DPAs can only effectively regulate and respond to the rise of LLMs by considering three key perspectives: the technology (technical analysis of how LLMs work), privacy risks and best practices. This article will present an overview of how LLMs work, outline the key takeaways and risks relating to data protection. 

Understanding LLMs 

LLMs are extremely large, complex machine learning systems capable of routinely generating highly articulate, plausible-sounding—but not necessarily true—linguistic content in response to queries on virtually any topic. LLMs consist of a large number of parameters (hundreds of billions or even trillions) organized across various architectural components, with each of them playing a specific role within the system. Some examples of LLMs are Chat GPT (OpenAI), Gemini (Google) and Bing Chat, which is now Copilot (Microsoft). 

LLMs typically employ a two-stage training procedure (with two substages: “pre-training” and “fine-tuning / alignment”) with multiple types of learning. In the pre-training stage, LLMs are typically trained on multiple datasets to enhance the models' adaptability to different contexts and scenarios. The model is trained on extremely large amounts of natural language, typically taken from aggregated sets of scraped websites and/or digitized books. For example, Common Crawl’s Data Infrastructure maintains a free, open repository of web crawl data that, on their own words: “can be used by anyone”. Other commonly used sources such are: Wikipedia, social media platforms such as Reddit and book corpora such as Project Gutenberg.  

The pre-training stage consists of a series of steps, applied repeatedly across batches of examples until a preset number of training cycles is reached. The second substage is fine-tuning, which is also divided into two (sub)stages: “supervised” learning, and “reinforcement” learning. During the “supervised” learning, the examples on which the model is trained are explicitly selected and curated by the developers, and, therefore, the amount of training data used at this stage is typically much smaller. During the reinforcement learning stage, a model will be trained by interacting in a dynamic environment with feedback, similar to a process of “trial and error.”  

Following the training procedure, the LLM moves the final stage: use. Here an interactive question-and-answer dialogue with an end-user will happen (by using prompts; for examples: asking questions, statements, or requests for information to the LLM). In this final stage, the adjustment of the degree of randomness of the responses generated by the LLM will be a key factor; therefore, a configuration variable (named “temperature”) will be used to enable greater control over the balance between randomness and determinism in the LLM's response. It is to be noted that in one of our previous posts, we explained how easy it was to manipulate ChatGPT.  

In view of the above, it is important for organisations and DPAs to understand each of these stages, as different risks may emerge during any of these. For example, hallucinations (instances of LLMs producing content detached from reality are commonly referred to as “hallucinations” or “confabulations”) may be triggered from prompts that specifically request false or misleading content, or, even, false criminal accusations against an individual may arise from the LLM after being asked a specific question.  

Privacy and Data Protection Risks associated with LLMs 

Due to their AI-enabled approach to automating linguistic tasks, LLMs raise a number of privacy and data protection risks. Among others, risks stem from design choices in the underlying technology while others may arise from practices relating to the processing of personal data. Furthermore, risks arise due to the inherent limitations in mathematical approaches to machine-based language acquisition and understanding, further supporting a need to understand the technological underpinnings of the software.  

As people become increasingly aware that any information about them online can be scraped and used in ways they cannot anticipate or control, they may post less information and withdraw from online discussions or communities, which implies that a chilling effect of free speech and expression online may arise. Furthermore, it is to be noted that most personal data that is publicly accessible still falls under data protection and privacy legislation in most jurisdictions. Data scraping undermines individual control over their personal data and takes their ability to control how their data is used; although individuals must be informed and able to exercise their personal data rights.  For example, individuals may not be able to correct out of date or inaccurate personal data if they are unaware it is held by an LLM developer. 

The paper also highlights tests demonstrating that LLMs can be tricked into revealing raw data contained in training datasets. This may create risks such as re-identification or de-anonymization, which create significant data protection concerns.  

Key Takeaway and Recommendations 

Considering the above, it is clear that a detailed understanding on a technological level is required to correctly evaluate the privacy risks for users and apply appropriate measures to protect their data. In this regard, the paper recommends that DPAs invest in building up technical know-how with respect to LLMs and AI systems in general. Organisations should be aware that what primarily differentiates LLMs, and generative AI from other forms of AI is, among others, the increase in scale of the data being processed and the complexity of the techniques used to develop and deploy the models. This recommendation can also be extended to organizations planning to develop or deploy any generative AI technology. Furthermore, these organizations should also consider the specific legislations for AI systems that may be applicable to them, by considering the use cases in which these technologies would be used and the risks that may arise from them.  

The paper also highlights an alternative LLM solutions that will go some way towards mitigating the above risks: Local LLMs. These types of LLMs are hosted locally on an organisation’s network. In these types of LLMs the processing of information is limited to interactions within the organisation, avoiding, in principle, transferring data to or its processing by any third party outside of the organisation’s network or ecosystem. This may have several advantages, such as having an isolated and private model, which may enhance the level of security and privacy as transfers of personal information to cloud or server based LLMs would not be needed. Therefore, personal data would not be subject to the terms of service of a commercial service. This solution comes with some challenges, such as barriers to use resulting from high demands on storage space and processing memory that a LLM usually requires.  

It is clear that the relevance and development of LLMs and AI in general is rapidly increasing so a detailed technological understanding is required by DPAs and organisations so that they can accurately evaluate privacy risks and mitigate them.  

Trilateral’s Responsible AI Services Team  have extensive experience supporting organisations to comply with their AI and data protection obligations, and assess any risk relating to these. They will assist you to get ready to develop and deploy compliant and responsible AI solutions. Please feel free to contact our advisors, who would be more than happy to help.