From Regex Redaction to NLP Redaction

In a previous article I covered a simple SQL regex approach for first-pass
redaction of customer comments. That approach is useful because it is
inexpensive, transparent and easy to run inside an existing analytics
pipeline.

But regex has a clear limitation: it only catches patterns that have been
explicitly defined.

Modern cloud NLP services go further. They use machine learning to identify
personal data even when it does not follow an obvious format.

The Three Broad Generations of PII Detection

1. Exact Pattern Matching

The simplest approach is regular expression matching.

[A-Z][a-z]+\s[A-Z][a-z]+
\S+@\S+\.\S+
\d+

This can catch obvious patterns such as:

• Names with capitalised first name and surname
• Email addresses
• Phone numbers
• Reference numbers

It is fast, inexpensive and deterministic. However, it is weak at
understanding context.

2. Dictionaries and Rules

The next layer combines regex with lookup lists and business rules.

For example, a system may maintain lists of:

• Common first names
• Towns and cities
• Company names
• Government departments
• Known identifier formats

This is stronger than regex alone, but still brittle. New names,
misspellings and ambiguous words can still cause problems.

3. Machine Learning Named Entity Recognition

Modern NLP services use machine learning models to identify entities in
free text.

This is normally called Named Entity Recognition (NER).

Tokenisation: Breaking Text into Units

The first stage is tokenisation. The service breaks a sentence into
smaller units that a model can process.

My name is David Hoffmann

This may become:

My
name
is
David
Hoffmann

Some modern models split words even further into subword tokens. For
example, a surname may be divided into smaller pieces if the model has
encountered it infrequently during training.

Embeddings: Turning Words into Numbers

Machine learning models do not understand text directly. They convert
words or tokens into numerical vectors called embeddings.

These vectors represent learned relationships between words.

The model does not store a simple list of meanings. Instead, it learns
statistical relationships from large volumes of training data.

Named Entity Recognition in Practice

NER models classify tokens or spans of tokens.

My name is David Hoffmann and my NI number is QQ123456C.

The model may return something like:

{
  "entities": [
    {
      "text": "David Hoffmann",
      "type": "PERSON",
      "confidence": 0.998
    },
    {
      "text": "QQ123456C",
      "type": "ID_NUMBER",
      "confidence": 0.963
    }
  ]
}

The important difference from regex is that the model can identify
David Hoffmann as a person because of context, not because
that exact name was hard-coded into a pattern.

Context Is the Main Advantage

Regex struggles with ambiguous words.

I bought an Apple laptop.
I ate an apple.

The same word appears in both sentences. A regex sees a string. An NLP
model uses the surrounding words to infer that the first use may refer to
an organisation, while the second refers to a piece of fruit.

Transformers and Attention

Many modern NLP services use transformer-based models. One of the key
concepts behind these models is attention.

Attention allows the model to evaluate relationships between words across
an entire sentence rather than simply reading from left to right.

David Hoffmann submitted his tax return.

The model can learn that his refers back to
David Hoffmann. This enables it to understand sentence
structure and context instead of treating each word independently.

Confidence Scores

Cloud NLP services usually return a confidence score alongside each
detected entity.

{
  "text": "David Hoffmann",
  "type": "PERSON",
  "confidence": 0.998
}

These scores allow organisations to apply different actions based on
confidence.

Why Cloud NLP Costs More Than Regex

A SQL regex replacement is inexpensive because it performs direct pattern
matching.

regexp_replace(comment, '\d+', '##')

An NLP service performs considerably more work before returning a result.

That additional computation is why NLP services are generally more
expensive than regex, but also substantially more flexible.

A Hybrid Redaction Pattern

In practice, the strongest approach is usually a layered one that combines
traditional pattern matching with machine learning.

This layered approach keeps processing costs lower while significantly
improving detection quality.

Where Large Language Models Fit

Large language models extend this idea further. Rather than simply
classifying tokens, they can reason across the meaning of an entire piece
of text.

John submitted feedback because he could not update his tax code after changing jobs.

A more advanced model may infer that:

• John is a person.
• he refers to John.
• tax code relates to a tax service.
• changing jobs provides contextual information about the user’s situation.

Rather than looking for individual patterns, the model attempts to
understand relationships between entities, actions and context across the
entire sentence.

Limitations of NLP Tools

NLP tools are not perfect. Like any statistical model, they can produce
both false positives and false negatives.

They may struggle with:

• Misspellings
• Very short comments with little context
• Slang and informal language
• Rare names
• Mixed languages
• Domain-specific terminology
• Ambiguous organisation names

Choosing the Right Approach

Regex and NLP should not be viewed as competing technologies. Each solves
a different part of the problem.

Conclusion

Regex remains an essential tool because it is fast, transparent and
inexpensive. It can remove many obvious identifiers directly within SQL
or an analytics pipeline with minimal overhead.

Cloud NLP services build on that foundation by introducing tokenisation,
embeddings, Named Entity Recognition, confidence scoring and
transformer-based contextual modelling. These capabilities allow them to
detect personal information that traditional pattern matching often
misses.

For analytics teams working with customer comments, survey responses and
other free-text datasets, the strongest solution is rarely regex or NLP
alone. A layered approach—using regex for obvious identifiers, NLP for
contextual detection and manual review for uncertain cases—provides the
best balance of cost, accuracy and governance.