How to Use Python for NLP and Semantic SEO: 4 Tested Scripts for Entities, Entity Gaps, Keyword Clustering and Search Intent

Python earns its place in semantic SEO when you have more text than you can read: hundreds of competitor pages, thousands of Search Console queries. Four jobs cover most of the work: extract the entities a page covers (spaCy), compare them with the pages that outrank you to find gaps, cluster keywords by meaning with sentence embeddings (sentence-transformers and scikit-learn), and label search intent in bulk. Each script below is under 40 lines and runs on a laptop. We tested them in October 2026 with spaCy 3.8, sentence-transformers 6.1 and scikit-learn 1.9.

Below: what NLP actually means in SEO, a five-minute setup, the four scripts with real output, when Google's own Natural Language API is worth paying for, and what you should not automate.

#What NLP means in SEO

Natural language processing (NLP) is the branch of AI that lets software work with human language: split text into tokens, reduce words to their base form, recognize named entities (people, brands, places, products), and turn meaning into numbers called embeddings, so that "cheap flights to Rome" and "budget airfare Rome" end up close together.

In SEO the term covers two different things.

What Google does with your text. Google's guide to its ranking systems lists several language systems. RankBrain relates words to concepts. Neural matching connects the concepts in a query with the concepts on a page. BERT reads how combinations of words change meaning. Passage ranking scores individual sections of a page. MUM, announced in 2021, is not used for general ranking, only for specific features. Google launched RankBrain in 2015, neural matching in 2018 and BERT in 2019 (Google), and in 2020 said its Knowledge Graph held "over 500 billion facts about five billion entities" (Google).

What you do with text, at scale. You run the same kinds of models on your pages, your competitors' pages and your query data, so you can see topics, entities and gaps instead of guessing them.

That second part is where Python fits into semantic SEO, the practice of optimizing for meaning (entities, relationships, intent) rather than exact-match keywords. Python doesn't make a page semantic. It measures what a page covers, faster than you can.

#What are "NLP terms" in SEO?

When a content optimization tool shows a list of "NLP terms", it means words and entities that often appear on the pages ranking for your keyword, extracted with an NLP model. They are a coverage hint, not a ranking factor: Google doesn't grade pages against a term list. Use them to spot subtopics you forgot, then cover those subtopics properly. The same goes for the older idea of LSI keywords.

#Set up Python for SEO NLP in five minutes

Create a virtual environment (python -m venv seo-nlp, then activate it) and install five things:

The small spaCy model (en_core_web_sm) is enough for SEO work. The transformer model (en_core_web_trf) recognizes entities more accurately but runs much slower. Older tutorials start with NLTK and Gensim. They still work, but you don't need either for the four jobs below.

#Script 1: extract the entities a page covers

Named entity recognition (NER) finds the proper nouns in a text and labels them: ORG for companies, PERSON, GPE for countries and cities, PRODUCT, EVENT, WORK_OF_ART, LAW. Counting them shows what a page is really about, which is often not what its title claims.

import requests, spacy
from bs4 import BeautifulSoup
from collections import Counter

nlp = spacy.load("en_core_web_sm")

def page_text(url):
    html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20).text
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "header", "footer", "aside", "form"]):
        tag.decompose()
    main = soup.find("main") or soup.find("article") or soup.body
    return " ".join(main.get_text(" ").split())

KEEP = {"ORG", "PRODUCT", "PERSON", "GPE", "EVENT", "WORK_OF_ART", "LAW"}

def entities(url, top=25):
    doc = nlp(page_text(url)[:200_000])
    counts = Counter(e.text.strip().lower() for e in doc.ents if e.label_ in KEEP)
    return counts.most_common(top)

for name, n in entities("https://en.wikipedia.org/wiki/Search_engine_optimization", 15):
    print(f"{n:>3}  {name}")

When we ran it on Wikipedia's SEO article in October 2026, the top entities were google (55 mentions), seo (22), pagerank (12), search (11), wikipedia (6), kinderstart (6) and matt cutts (3). Two lessons from that list. The model finds the real topic anchors: Google, PageRank, Matt Cutts, the KinderStart lawsuit. It also returns noise such as "pdf" and "learn", so read the output and keep a short stoplist. Dates, money and numbers are left out on purpose because they rarely tell you what a page is about.

#Script 2: find entity gaps against the pages that outrank you

An entity gap is an entity that most top-ranking pages mention and yours doesn't. It's the semantic version of a content gap analysis, and it reuses the entities() function from script 1.

from collections import Counter

my_url = "https://example.com/your-page"
top_urls = [
    "https://competitor-one.com/page",
    "https://competitor-two.com/page",
    "https://competitor-three.com/page",
]

mine = {name for name, _ in entities(my_url, top=60)}
coverage = Counter()
for url in top_urls:
    coverage.update({name for name, _ in entities(url, top=60)})

gaps = [(name, n) for name, n in coverage.most_common() if n >= 2 and name not in mine]
for name, n in gaps[:25]:
    print(f"{name}: on {n} of {len(top_urls)} top pages, missing from yours")

Take the top three to five organic results for your main query, skipping forums and video pages, which follow different rules. An entity on two of three pages is a candidate subtopic. Don't paste it into your page. Ask why it matters to the searcher, and add a section if it does.

Two practical notes. Some sites block scripted requests with a 403; for those, paste the text in manually or use a crawler that can extract page text. And entities are only half of a gap: a competitor can also answer a question you skip without naming any entity, so read the pages too. Our guide to competitor website analysis covers the rest.

#Script 3: semantic keyword clustering in Python

Keyword clustering groups queries that one page should answer, so you write one strong page instead of five that compete with each other. Matching on shared words fails as soon as people phrase things differently. Sentence embeddings place queries with the same meaning close together even when they share no words, and a clustering algorithm groups the close ones. (For the concept and the non-code options, see what keyword clustering is.)

Export your queries first: in Search Console, open Performance → Search results, set the date range, click Export → Download CSV, and use the Queries.csv file. It has the columns Top queries, Clicks, Impressions, CTR and Position.

import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.cluster import AgglomerativeClustering

df = pd.read_csv("Queries.csv")
keywords = df["Top queries"].tolist()

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(keywords, normalize_embeddings=True)

clusterer = AgglomerativeClustering(
    n_clusters=None, metric="cosine", linkage="average", distance_threshold=0.3
)
df["cluster"] = clusterer.fit_predict(embeddings)

# Name each cluster after its highest-impression query
names = df.sort_values("Impressions", ascending=False).groupby("cluster")["Top queries"].first()
df["topic"] = df["cluster"].map(names)

summary = (
    df.groupby("topic")
    .agg(queries=("Top queries", "count"), impressions=("Impressions", "sum"), clicks=("Clicks", "sum"))
    .sort_values("impressions", ascending=False)
)
summary.to_csv("clusters.csv")
print(summary)

#What the threshold does (real output)

We ran this exact script on the 15 queries this article received in Search Console over the 90 days to October 2026. The only change between runs was distance_threshold:

Start around 0.3 with all-MiniLM-L6-v2 and check a sample by hand. Too low and every keyword becomes its own cluster. Too high and different intents merge. Here the 0.3 result also explains this page's structure: the same URL gets both "how to use Python for semantic SEO" and "what NLP means in SEO" searches, so it answers both.

For lists above roughly 10,000 keywords, agglomerative clustering gets slow and memory-hungry because it compares every pair. Switch to sentence_transformers.util.community_detection, which the library provides for large sets. The more accurate (and more expensive) alternative is SERP-overlap clustering: group keywords whose top 10 results share several URLs. That needs a SERP data source, which our guide to APIs in SEO compares.

#Script 4: label search intent in bulk

Search intent is the goal behind a query: learn something, compare options, buy, or reach a specific site. A rules-based labeler is crude, but it is transparent and handles most queries in a Search Console export in under a second.

import re
import pandas as pd

RULES = [
    ("transactional", r"\b(buy|price|pricing|cost|cheap|discount|coupon|deal|near me|hire)\b"),
    ("commercial", r"\b(best|top|vs|versus|review|reviews|alternative|alternatives|compare)\b"),
    ("informational", r"\b(how|what|why|when|who|guide|tutorial|example|examples|meaning|ideas)\b"),
]

def intent(query):
    q = query.lower()
    for label, pattern in RULES:
        if re.search(pattern, q):
            return label
    return "navigational or unclear"

df = pd.read_csv("Queries.csv")
df["intent"] = df["Top queries"].apply(intent)
print(df.groupby("intent")["Impressions"].sum().sort_values(ascending=False))

Order matters: "best price for X" should count as transactional, so that rule runs first. Add your brand name to a navigational rule. For queries the rules miss, send batches of about 100 to an LLM API with a fixed list of labels and one example per label, then spot-check a few dozen results by hand before trusting the rest.

Pair the intent column with the cluster column from script 3 and you have a content plan: one row per topic, with its total impressions and the format it needs (guide, comparison, product page).

#Google's Natural Language API: entities the way Google labels them

The Cloud Natural Language API is the closest you can get to seeing entities through Google's eyes. Its entity analysis returns each entity with a type, a salience score (0 to 1, how central the entity is to the whole text) and, for well-known entities, a Knowledge Graph machine ID (mid) and a Wikipedia URL.

from google.cloud import language_v1  # pip install google-cloud-language

client = language_v1.LanguageServiceClient()  # needs Google Cloud credentials

def google_entities(text, top=15):
    doc = language_v1.Document(content=text, type_=language_v1.Document.Type.PLAIN_TEXT)
    resp = client.analyze_entities(request={"document": doc, "encoding_type": language_v1.EncodingType.UTF8})
    for e in sorted(resp.entities, key=lambda e: e.salience, reverse=True)[:top]:
        print(f"{e.salience:.3f}  {e.name}  ({language_v1.Entity.Type(e.type_).name})  {e.metadata.get('mid', '')}")

google_entities(page_text("https://example.com/your-page"))

Use language_v1 on purpose: the v2 API dropped the salience field and the Wikipedia URL metadata (release notes).

Cost, as of October 2026: billing is in units of 1,000 characters. Entity analysis is free for the first 5,000 units a month, then $1.00 per 1,000 units up to a million (pricing). A 2,000-word article is about 12,000 characters, or roughly 12 units, so the free tier covers a few hundred articles a month.

One caveat matters more than the price. This public API is not the system Google Search uses to rank pages. If your target entity has low salience on your own page, the page probably isn't clearly about it. That's a useful sanity check, not a ranking score.

#Which Python NLP tool fits your SEO task?

#Semantic SEO automation: what to automate and what not to

Automate the measuring. Entity extraction, clustering, intent labels and internal link suggestions (compare page embeddings and link pages that sit close together) are repetitive, and scripts do them better than people.

Don't automate a page for every phrasing. Google's guide to optimizing for generative AI search says that creating separate content for every variation of a query, fan-out queries included, mainly to manipulate rankings violates its scaled content abuse policy. Clustering exists to merge variations into one page, not to multiply pages. That is also how topical authority gets built: by covering distinct subtopics well, not by publishing near-duplicates.

If what you actually want is published articles rather than analysis, that is the part Outrank automates: keyword research with live search volume and difficulty, one long-form article a day with internal links and an FAQ, published straight to WordPress, Webflow, Shopify, Framer, Wix, Notion, Ghost or a webhook. In the 90 days to October 2026 it wrote 200,871 articles for 4,348 sites, and 63% of them targeted keywords with a difficulty under 20 (Outrank platform data). The scripts above remain the right tool for auditing what you already have.

#FAQ

#What is NLP in SEO?

NLP in SEO means using natural language processing to understand text the way search engines do: by its entities, topics and intent rather than its exact keywords. Google uses language systems such as RankBrain, neural matching and BERT in ranking. SEOs use NLP libraries to measure what pages cover and how queries group together.

#How do you use NLP in SEO?

The four most useful applications are extracting the entities a page covers, comparing those entities with top-ranking pages to find gaps, clustering keywords by meaning so each topic gets one page, and labeling search intent at scale. All four can be done with free Python libraries (spaCy, sentence-transformers, scikit-learn) on a Search Console export.

#Can you use Python for SEO?

Yes. Python is widely used in SEO to pull Search Console data through its API, crawl and parse pages, cluster keywords, extract entities and automate reports. You don't need to be a developer: the scripts in this guide are under 40 lines each and run on a normal laptop.

#What is the best Python library for NLP in SEO?

For entity extraction, spaCy: it's free, fast and accurate enough with its small English model. For keyword clustering and semantic similarity, sentence-transformers with scikit-learn. If you want entities labeled with salience and Knowledge Graph IDs, Google's Cloud Natural Language API is the paid option.

#What is semantic keyword clustering?

Semantic keyword clustering groups search queries by meaning rather than by shared words, so queries one page should answer end up together. In Python you embed each query with a sentence-transformers model and group the vectors with agglomerative clustering, using a cosine distance threshold (around 0.3 is a good starting point).

#Does Google use NLP for ranking?

Yes. Google's ranking systems guide lists several language systems used in Search, including RankBrain, neural matching, BERT and passage ranking. Google's public Natural Language API is a separate product, so its scores show how a model reads your page, not how Google ranks it.