Outgrace, home

How GDPR treats automated decisions in customer profiling

The sources behind this explainer confirm the GDPR's structure and balancing principle, but not the text of Article 22. Here is what leaders can and cannot yet rely on.

Leaders who profile customers need to know when the GDPR's rules on automated decisions apply to their systems. The sources behind this article cannot tell them. They establish the Regulation's structure, its purpose and the balance it strikes between privacy and other rights. They also offer a useful comparison from outside Europe. They do not reproduce Article 22, the provision on automated decision-making that this topic turns on, so this piece cannot tell you what safeguards it requires. What follows explains what can be stated with confidence, what cannot, and how to close the gap before a board relies on a view about it.

How it works

Regulation (EU) 2016/679, the General Data Protection Regulation, was adopted on 27 April 2016 and published in the Official Journal as L 119 on 4 May 2016, according to the EUR-Lex database. The publication runs from page 1 to page 88 of that issue, and the Regulation replaces Directive 95/46/EC. Its full title covers the processing of personal data and the free movement of such data. Both halves matter. The Regulation governs how individuals' data is handled and also how that data moves, which is why it is read as a rule for the single market as well as for privacy.

The recitals, the explanatory preamble that comes before the operative articles, describe the protection of personal data as a fundamental right. According to the EUR-Lex text, they cite Article 8(1) of the Charter and Article 16(1) of the Treaty on the Functioning of the European Union. The same recitals say the right is not absolute. It has to be weighed against other fundamental rights, and they list freedom to conduct a business and the right to an effective remedy and fair trial among the rights the Regulation respects. For a leader, that is the starting point for any profiling question. The customer's right sits on a scale alongside the company's own rights, and the recitals expect the two to be balanced rather than one simply prevailing.

The recitals also explain why the rules exist. They point to rapid technological development and to a significant increase in the scale of personal data collection and sharing. They note that private companies and public authorities use personal data on an unprecedented scale, that cross-border flows of personal data within the internal market have substantially increased, and that the exchange of personal data between public and private actors across the Union has grown. Customer profiling draws on several of these trends at once: collection, sharing and movement across borders. A score built from data gathered in one country and used in another falls squarely within the concerns the recitals describe.

The gap is in the provision itself. The excerpt supplied for this research contains no Article 22 text, so this article cannot confirm what that provision says about automated decisions, human review, explanation or objection. The EUR-Lex page also states that the act 'has been changed' and gives the consolidated version date as 04/05/2016. The text on that page is therefore not guaranteed to be the current consolidated version, and it must be checked before anyone cites it. Until someone has read the current text, every statement about Article 22 in internal papers should be marked as unverified. That includes any summary a vendor or consultant has passed to you.

Risk scores the Public Safety Assessment produces, per Brookings (US criminal justice)
3
Fewer defendants detained in a simulated checklist model, per Brookings (Jung et al.)
30%
Range of judges' pretrial release rates in a simulation, per Brookings (Jung et al.)
50% to 90%

Who it affects

The fact base identifies the group affected in general terms: EU-based or EU-facing organisations that process personal data. It names no company, sector or region in connection with customer profiling. The phrase EU-facing matters for any company headquartered outside Europe, but the sources do not define it. Ask counsel to define it against your own customer base before you assume you are out of scope. A leader running a customer-scoring system in any sector should treat that question as open until it has an answer on file.

The only sector-specific material in the sources concerns US criminal justice. Brookings reports that risk assessment instruments predict a defendant's future risk and inform judicial decisions such as pretrial incarceration. That work is not about the GDPR, and it should not be read as a verdict on any commercial system. Its value to a profiling team lies in the questions it raises. The European Data Protection Board, the EU body whose homepage the sources cite, states its aim as equal data protection rights across Europe, achieved through clear guidance and consistent enforcement. The EDPB also says strong rules keep individuals in control of their data, protect personal freedom, autonomy and freedom of expression, help prevent misuse, unauthorised access, identity theft and manipulation, and build trust in online services. These are stated purposes, and they explain why regulators care about profiling. They do not tell you what the rules require. The EDPB homepage excerpt names no Article 22 and no profiling guidance, and the sources contain no EDPB guidelines on automated decisions. Do not cite EDPB guidance on this topic until you have located the document yourself.

What leaders get wrong

The errors below are the ones to test first, because each is an assumption rather than a finding. Each is cheap to check, and each tends to surface only when a customer or regulator asks a question the team cannot answer.

  • Assuming a person in the loop settles the matter. Whether human involvement counts depends on what the provision requires, and the sources here do not say. Find the text before you design the process around it.
  • Reading the Brookings evidence as GDPR guidance. The report covers US criminal justice. Its use for a customer-profiling team is in the questions it raises, not in any rule it reports.
  • Accepting a trade-secret answer as the end of the conversation. Brookings reports that commercial vendors often hide risk assessment details behind trade-secret claims, and that critics raise the absence of transparency as a concern. In our experience, a vendor's refusal to explain its model is a fact about your procurement. It does not answer a customer's question or a regulator's.
  • Treating a score as an input when in practice it is the decision. Brookings notes that judges can disregard risk assessment release recommendations when they seem too strict or too lax. In Loomis v. Wisconsin (2016), the Wisconsin Supreme Court said the judge's decision was not solely determined by the COMPAS tool, according to Brookings. The lesson for any firm is to know, and record, whether a s

What to do about it

Start with an inventory, not a legal opinion. List every system that scores or ranks customers, what each score feeds, who acts on it, and whether anyone reviews the result before it reaches the customer. In our experience, the common finding is a score that everyone calls advisory and that in practice is never overridden. Finding that out is the point of the exercise, and it costs little compared with discovering it during a complaint.

Then get the current text and the legal judgement. Have counsel read the consolidated GDPR, with Article 22 and the surrounding recitals, against your inventory. Put three questions to them: which of our scores feed a decision about an individual, what the Regulation requires in that case, and what we must be able to explain to a customer who asks. Whether a given system falls within the provision is a question for a lawyer or your data protection lead. This article cannot answer it, and neither can a vendor's marketing sheet.

Test explanation before anyone asks for one. Could a trained member of staff describe, in plain language, the main factors behind a score and the reason for a particular decision? If not, the problem is in the design, not the disclosure. Ask vendors what they can share under contract, and record what they decline to share. The Brookings report shows why that record matters. Where a model's details are hidden, outsiders cannot easily test it, and critics cite the possibility of bias as a reason to insist on transparency.

Measure human review rather than assume it. Brookings reports a simulation in which judges' pretrial release rates varied from roughly 50% to almost 90%. It also reports a simulated checklist model that would have detained 30% fewer defendants, with no corresponding rise in pretrial misconduct. Those results concern courts, not customers, and they are simulations. The lesson for a commercial team is narrower. If human decisions vary this much, a review step that never changes an outcome may not be the safeguard it appears to be. Track how often reviewers override a score, and record why.

Keep the record. A board that is later asked what it knew should be able to point to the inventory, the legal advice, the vendor disclosures and the override log. Each of those documents is cheap to produce now and expensive to reconstruct after a complaint.

The right to the protection of personal data is not an absolute right; it must be considered in relation to its function in society and be balanced against other fundamental rights, in accordance with the principle of proportionality.

Regulation (EU) 2016/679, recitals, eur-lex.europa.eu

Newsletter

The Briefing

One email a week: the leaders we profiled, the ideas worth stealing, and nothing else.

Weekly, free, one click to leave. Read our privacy policy.