NLP Basics
Python · Advanced Python & AI
Theory
Natural Language Processing (NLP) lets computers work with human text. Models need numbers, so the first job is turning text into vectors. A typical classic pipeline is: clean the text (lowercase, strip punctuation), tokenize it into words, optionally remove stop words like 'the', then vectorize. CountVectorizer counts how often each vocabulary word appears in each document (bag of words).
TfidfVectorizer goes further with TF-IDF (term frequency x inverse document frequency): words frequent in one document but rare across the collection get higher weights, while words found everywhere get lower ones, and each row is normalised to length 1. For ['great service', 'poor support'] the vocabulary is great, poor, service, support, and each document gets 0.707 for its two words. These vectors can then feed a classifier such as logistic regression for sentiment analysis.
Why we use it
Huge amounts of business data are text: reviews, tickets, emails, chats. NLP turns it into numbers so you can classify sentiment, route tickets, find topics or search documents automatically.
When to use it
TF-IDF with a linear model is a fast, strong baseline for text classification and keyword search. For tasks needing context and meaning (summaries, question answering), transformer-based models usually perform far better.
How it works
- Raw texts
- Clean: lowercase, remove punctuation
- Tokenize into words
- Vectorize with TfidfVectorizer: one column per word
- Have labels (e.g. sentiment)? Yes: Train a classifier on them. No: Cluster or search the vectors.
- Predictions or insights from text
Syntax
from sklearn.feature_extraction.text import TfidfVectorizer
X = TfidfVectorizer().fit_transform(texts)Example
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(['great service', 'poor support'])Key points
- TF-IDF converts text into numeric feature vectors
- Modern NLP increasingly uses transformer models instead of TF-IDF
- Vocabulary columns are sorted alphabetically (get_feature_names_out)
- Fit the vectorizer on training texts only
- Words unseen during fit are ignored at transform time