What is Git?

Git is a distributed version control system (DVCS) specifically design to manage and track changes in a project's codebase over time. It allows multiple developers working on the same project to maintain their own complete history of changes, providing a comprehensive record of who made what changes and when. This system is like a "time machine" for your code, enable developers to easily revert to previous versions if needed, compare different history of the project, and collaborate efficiently without overwriting each other's work. Git's distributed nature means that every developer has a full copy of the entire project history on their local machine, ensuring that the project is prevented against data loss and allow seamless collaboration across diverse teams.

Why Git is Used?

Git solves many real problems developers face while building software in a team (or even solo).

1. Track Changes: See what changes are made in which files, when, and by whom.

2. Undo Mistakes: Safely undo any mistaken changes made to any file.

3. Work in Parallel: Multiple developers can create branches to work on different features simultaneously without waiting for each others.

4. Merge Code: Combine code from different developers.

5. Detect Conflicts: While merging, Git helps track conflicts in the code and resolve them.

6. Maintain a Remote Repository: Keep a remote repository for the codebase on a local or cloud server for backup and deployments.

Git Basics and Core Terminologies

• Repository

  A repository (or repo) is a virtual storage space for managing and storing digital assets like a codebase, data, or project files. At its core, a Git repository is the hidden .git directory located in the root of your project folder. Read this blog to learn more about the .git folder.

• Commit

  A commit in Git is a snapshot of your project at a specific moment. Essentially, it saves the project's history at a particular time, including the staged changes and metadata.

• Staged changes

  These are modifications in your project's files that have been marked in their current version to be included in the next commit. This is done using the git add command.

• Branch

  A branch is like a separate workspace created from a specific commit in a repository where you can make changes. It can be merged with another branch as a commit or kept as a separate branch.

• HEAD

  HEAD points to the current commit we are working on in a specific branch. When you make a commit, it is added on top of HEAD. When you checkout a branch, HEAD moves to that branch.

• Checkout

  Checkout in Git is a command used to move HEAD to a different branch or commit and update your working files accordingly.

Common Git Commands

• Initialize a new Git repository:
  git init

• Copy an existing remote repository to your machine:
  git clone <url>

• See the current state of files in the repository:
  git status

• Stage file changes for the next commit:
  git add <filename> → This stages specific file for the next commit
  git add . → This stages all file changes in the repository for the next commit.

• Save/commit staged changes to the repository history:
  git commit -m “message here”

• Display the commit history:
  git log

• Combines another branch into the current branch.

  git merge <branch>

• Show differences between file versions:
  git diff → This show changes we’ve made in the files that haven't yet added to the staging area with git add .
git diff <branch1> <branch2> → This displays all changes that are in branch2 but not in branch1.

  git diff <commit-id1> <commit-id2> → We can use commit hashes to see the differences between any two points in project's history.

• Lists all branches in the repository:
  git branch

• Create a new branch:

  git branch <branch-name>

• Switches to another branch or commit:

  git checkout <branch/commit hash>

• Create and switch to a new branch:

  git checkout -b <branch>

• To add a remote repository in git repo:

  git remote add <remote-name> <remote-url>

  replace <remote-name> with a name for the remote (e.g., origin) and <remote-url> with the URL of remote location (eg. GitHub or GitLab).

• Upload local commits to the remote repository:

  git push

• Fetch and merge changes from the remote repository:

  git pull

I recently started digging deeper into RAG (Retrieval-Augmented Generation) and realized that while the basic RAG architecture is powerful, it’s also far from perfect. So, in this article, let me explain:

• How basic RAG works

• Why RAG struggles sometimes

• Different optimization techniques to improve accuracy

• When not to overengineer things

How Basic RAG Works

At its core, a RAG system does something simple:

1. Take user input → a query or question.

2. Convert it into vector embeddings → numerical representations of meaning.

3. Search the vector database → e.g., Qdrant, Pinecone, or FAISS.

4. Retrieve relevant chunks of information.

5. Send the retrieved chunks + user query to an LLM.

6. LLM generates an answer using both its knowledge + provided context.

Sounds neat, right? But here’s the problem…

The Garbage In, Garbage Out (GIGO) Problem

RAG is only as good as the input you give it.
If the user’s query is vague, incomplete, or inconsistent, the retrieved context may not match well, leading to poor answers.

For example:

• Your vector DB has chunks about “machine learning model deployment”

• The user asks: “How to put my AI online?”

• The retriever might miss relevant chunks because the wording doesn’t match, even though the intent is related.

So, we need smarter techniques to bridge this gap and make RAG more accurate.

Ways to Make RAG Smarter

1. Query Rewriting (Simplest Fix)

Idea:
Before hitting the vector DB, rewrite the user’s query to make it more clear, structured, and context-friendly.

Flow:

How it helps:

• Better embeddings → better chunk retrieval

• More consistent matches with your knowledge base

When to use it:

• Works great for small optimizations

• Minimal performance impact

2. Multi-Query Retrieval (More Accurate, Slightly Slower)

Idea:
Instead of one improved query, generate multiple related queries to cover all possible angles of the user’s intent.

Flow:

Why it works:

• Covers semantic variations the original query might miss

• Retrieves more complete and accurate context

• Significantly improves overall precision

Trade-off:

• Increases retrieval time slightly

• Best for complex or ambiguous queries

3. HyDe Approach (Hypothetical Document Embeddings)

This one’s clever. Instead of directly searching the vector DB with the user’s query, we:

1. Generate a “hypothetical answer” using an LLM.

2. Convert this generated answer into vector embeddings.

3. Use those embeddings to search the vector DB.

4. Retrieve highly relevant chunks.

5. Finally, send the best chunks + user query to the LLM for final output.

Flow:

Why it works:

• The LLM “imagines” the right answer first

• This makes the retrieval process much more accurate

• Especially useful when user queries are vague or incomplete

Bonus: Combine Multi-Query + HyDe = Ultra Accuracy

For critical tasks where accuracy matters more than speed, you can combine techniques 2 and 3:

• Use HyDe to generate a better search base

• Then perform multi-query retrieval

• Finally, pick the highest-frequency chunks for the final answer

This gives you near-perfect retrieval accuracy, but it’s slower — so use it wisely.

Final Thoughts

The key takeaway here is:

RAG isn’t broken — it just needs help understanding what you really mean.

• Use query rewriting for quick wins

• Use multi-query retrieval when precision matters

• Use HyDe for vague queries or weak context

• Combine techniques only when necessary

And most importantly:

Don’t overengineer your RAG pipeline to kill a cockroach
Keep it simple unless your use case truly demands ultra accuracy.

Maybe it made up a fake reference…
Maybe it skipped something important…
Or maybe it confidently told you something completely wrong.

Well, if you’re working with Retrieval-Augmented Generation (RAG) systems, these problems are even more common. RAG sounds powerful — combine an LLM with an external knowledge base — but in reality, most RAG pipelines break in subtle ways.

Don’t worry, though. In this article, I’ll explain:

• Why RAG systems fail

• The 5 most common failure cases

• How to fix them quickly

• Best practices to make your RAG pipelines more accurate and reliable

Let’s dive in.

Poor Recall → Missing the Right Content

Imagine you ask your RAG-powered chatbot:
"What are the eligibility criteria for the new AWS Activate program?"

And it replies:
"Sorry, I couldn’t find anything relevant."

That’s poor recall — your retriever didn’t fetch the right context.

Why it happens

• Your knowledge base isn’t updated.

• Indexing missed some documents.

• Query expansion is weak.

Quick Fixes

• Enrich & update your knowledge base → Keep your database fresh.

• Human-in-the-loop reviews → Get experts to validate coverage gaps.

• Query expansion → Add synonyms and related terms for better hits.

Bad Chunking → Broken Context

Chunking is how you split your documents before indexing.
Do it wrong, and your RAG system either:

• Misses important context, OR

• Fetches too much irrelevant data, confusing the model.

Why it happens

• Splitting blindly by token count.

• Ignoring semantic boundaries like paragraphs or sections.

Quick Fixes

• Semantic chunking → Break at logical boundaries.

• Dynamic chunk sizing → Adjust based on document structure.

• Hybrid retrieval → Use both dense embeddings (concept-based) + sparse retrieval (keyword-based).

Tip: Don’t just feed RAG random pieces of text. Make sure your chunks carry meaning.

Query Drift → The Model Loses the Plot

Sometimes your retriever rewrites queries to improve results…
But in doing so, it changes the meaning of your question.

For example:
User query: “Show me the top 5 fastest-growing AI startups in India.”
Retriever reformulation: “AI startups India revenue report.”

Suddenly, you’re getting financial reports instead of growth data.

Quick Fixes

• Controlled query rewriting → Expand queries but keep intent intact.

• Context adherence checks → Track how much reformulated queries deviate.

• Prompt engineering → Use clearer, tighter instructions for the retriever.

Outdated Indexes → Stale Knowledge

RAG systems fail badly in recent events.
Ask it about OpenAI’s latest model release, and it might give you data from 2022.

Why it happens

• Indexes aren’t updated frequently.

• No metadata on document freshness.

Quick Fixes

• Automate index updates → Schedule frequent rebuilds.

• Add versioning & timestamps → Track when data was last updated.

• Automated fact-checking → Flag outdated or inconsistent answers.

Hallucinations → The LLM Makes Stuff Up

Even with RAG, models sometimes invent facts that don’t exist anywhere.
Why? Weak or irrelevant context.

Example:
"Who founded SpaceX?"
RAG retrieves nothing useful → LLM hallucinates:
"It was founded by Steve Jobs in 2010."

Quick Fixes

• Better retrieval + reranking → Ensure high-quality, relevant chunks.

• Structured output formats → Force models to stick to facts.

• Continuous context optimization → Improve query expansion + filtering.

Quick Summary

Final Thoughts

RAG is powerful — but fragile.
Most failures happen before generation — at the retrieval and chunking stages.

If you:

• Keep your indexes fresh

• Use smart chunking

• Control query rewriting

• Tune retrieval + reranking

…your RAG system instantly becomes 10× more reliable and much harder to break.

In short:

Good RAG ≠ Good LLM.
Good RAG = Good Retrieval + Good Generation + Good Context.

“Who won the IPL 2024 finals?”

…and it confidently gave you the wrong answer?

That happens because most AI models, including GPT, don’t actually know everything. They’re trained on huge amounts of data, but their knowledge is frozen at the time of training. If you ask about recent events or company-specific data, they might hallucinate — meaning they make things up.

Now imagine this instead:

• You have your own knowledge base (a large source of information)

• AI first searches in your database

• Then it understands the context

• Finally, it generates a smart, relevant answer

That’s exactly what Retrieval-Augmented Generation (RAG) does.
It bridges the gap between an AI model’s training data and your real-world, up-to-date information.

Why Do We Need RAG?

Think of a library.

• GPT is like a librarian who has read millions of books.

• But the librarian can’t remember everything perfectly.

• Sometimes, you want fresh information or specific documents that aren’t in their memory.

RAG acts like giving the librarian a catalog system:

• First, they search the right shelf (retrieval)

• Then, they summarize and explain (generation)

This makes AI:
More accurate
More reliable
More context-aware
Perfect for real-time knowledge

How RAG Works (Retriever + Generator)

Let’s break it into two main components:

Step 1 — Retriever 🔍

• Think of it like Google Search for your knowledge base.

• It finds the most relevant documents based on your query from the Data Source.

• Uses vector embeddings to compare meaning, not just keywords.

For example:

You ask: “How to install Ubuntu on Raspberry Pi?”

• Retriever looks into your docs/wiki

• Finds the most relevant guides

• Sends them to the generator

Step 2 — Generator ✍️

• This is your LLM (e.g., GPT, Claude, Gemma).

• It reads the retrieved documents and uses them to create an accurate, human-like answer.

Example answer:

“To install Ubuntu on a Raspberry Pi, download the Ubuntu Server image, flash it using Raspberry Pi Imager, insert the SD card, and boot your Pi. Make sure to enable SSH if needed.”

Quick Example Flow

You ask: “Who is the CEO of OpenAI?”

• Retriever: Searches your knowledge base → finds a doc saying “Sam Altman is the CEO.”

• Generator: Reads it → gives you a natural reply:

“The current CEO of OpenAI is Sam Altman.”

What is Indexing?

Before AI can retrieve anything, we need a searchable structure. That’s where indexing comes in.

Think of indexing like a table of contents in a book:

• It breaks your documents into chunks

• Converts them into vectors (we’ll get there in a sec)

• Stores them in a vector database like Pinecone, Weaviate, Milvus, or FAISS

• When you search, AI compares your query vector to these stored vectors and fetches the closest matches.

Why We Perform Vectorization?

Normal keyword search sucks for AI. Why?

• If you search “AI laws”, a normal search engine might skip documents that say “legal regulations for artificial intelligence.”

• But AI needs meaning, not exact words.

That’s why we use vector embeddings:

• We convert text → numerical vectors in a high-dimensional space.

• Sentences with similar meaning end up closer together.

• This makes retrieval semantic instead of keyword-based.

Example:

• “Install Ubuntu on Pi” → Vector A

• “Setup Raspberry Pi with Ubuntu” → Vector B

• A & B are close in vector space → retriever understands both are related

Why Do RAGs Exist?

We created RAG because LLMs alone aren’t enough:

• They forget private, domain-specific knowledge

• They hallucinate when uncertain

• They can’t access real-time data

• They don’t know your internal documents

RAG lets you connect AI to your data safely, without retraining the whole model.
That’s why companies, chatbots, SaaS platforms, and knowledge assistants rely on RAG.

Why We Perform Chunking

Imagine dumping a 500-page PDF into ChatGPT.
It would struggle to find the relevant parts efficiently.

That’s why we split documents into smaller pieces → called chunks.

• Typical chunk size = 300 to 800 tokens

• Each chunk is indexed separately

• This makes searching faster and more accurate

Why Overlapping is Used in Chunking

Sometimes, the important context lies between two chunks.

Example:

• Chunk 1 ends with: “The API key should be stored securely.”

• Chunk 2 starts with: “Never commit secrets to GitHub.”

If we don’t overlap, AI might miss the connection between them.

That’s why we use sliding windows:

• Each chunk shares some sentences with the previous one

• Ensures AI always has full context

Final Thoughts

Retrieval-Augmented Generation (RAG) is like giving your AI Google + Brain Power:

• Retriever → finds the right knowledge

• Generator → writes smart answers

• Indexing + Vectorization → make search semantic

• Chunking + Overlap → make results accurate

If you’re building:

• AI-powered chatbots 🤖

• Document assistants

• Knowledge search systems

• Customer support bots

…you’ll definitely need RAG.

Quick Summary

Now imagine we could give AI a “toolbox” – like a set of apps or functions – and let it pick the right one depending on the task. That’s where Agentic AI comes in.

What are AI Agents?

Think of an AI agent as not just a chatbot, but like a person who can think, plan, and use tools to get things done.

• A normal AI LLM model just predicts text based on what you give it.

• An agentic AI model takes it further: it reasons step by step, decides what to do, uses tools, and then gives you the final answer.

It’s like the difference between a student who memorizes formulas vs. one who knows how to apply formulas, use a calculator, and solve real problems.

How Agents Work

Here’s the flow:

1. You ask a question → “What’s the weather in Delhi tomorrow?”

2. AI checks its toolbox → “I don’t know live weather, but I see a weather API tool available.”

3. AI decides the step → “Use the weather API with location=Delhi.”

4. Tool runs and returns data → “Sunny, 34°C.”

5. AI explains back to you → “It’ll be sunny in Delhi tomorrow with a high of 34°C.”

So the AI doesn’t magically know the weather. It just knows how to pick the right tool and use it.

The Role of Tools

Tools are functions or APIs we expose to the AI.

Example:

{
  "tools": {
    "calculator": (expression) => eval(expression),
    "weather": (city) => getWeather(city),
    "dbSearch": (query) => queryDB(query)
  }
}

Now when AI sees “27 × (32 + 67) – 93 ÷ 45” it knows:

• Use the calculator tool.

• Parse the expression.

• Return the answer.

If you ask about sales data, it can call dbSearch. If you ask about the weather, it calls weather.

The AI itself doesn’t do the math or fetch live info – it delegates the task.

Why Agentic AI is Powerful

• Flexibility → The Same model can do many tasks if given the right tools.

• Scalability → Add/remove tools without retraining the model.

• Reliability → Tools return exact results, AI just interprets.

• Human-like reasoning → The AI acts like an assistant that knows when to Google, when to calculate, and when to just answer directly.

Real-World Examples

• ChatGPT with Browsing → When you ask about current events, it calls a search tool.

• LangChain Agents → Define multiple tools (search, calculator, database) and let the model pick.

• Copilot for Devs → Calls code search, compiler, or documentation functions.

Wrapping Up

Agentic AI is not just about “chatting.” It’s about thinking + acting + using tools.
Just like we don’t solve everything with memory, AI shouldn’t either. We check Google, we use calculators, we read docs. Agents do the same – they just need to know what tools are in their kit.

So, the future of AI is not just bigger models – it’s smarter agents with the right tools.

Next time you use an AI, think: Is this just a chatbot, or is it an agent using tools behind the scenes?

If you ask it to write code, it won’t.
Why? Because it’s not trained for that — it’s following patterns, not reasoning.

Most language models work like this:

• Input → Predict most likely text → Output.
  No planning. No deep thought. Just autocomplete on steroids.

But as developers, we can actually programmatically force the model to reason step-by-step.
That’s where Chain-of-Thought (CoT) comes in — not as a “prompt trick,” but as part of your system design.

The Problem: LLMs Don’t Think By Default

LLMs are trained to complete text, not to break problems into logical substeps.
When given:

20 + 32 × 67 + 93 - 267 ÷ 45

They might jump straight to an answer.
If they make a small mistake early, the final answer is wrong — and you won’t even know why.

The Solution: Force Thinking with START → THINK → EVALUATE → OUTPUT

Instead of just asking for the solution to a complex problem
We define a protocol that the AI must follow:

1. START — Understand the problem.

2. THINK — Break it into smaller steps.

3. EVALUATE — Wait for a human or another AI to check the step.

4. OUTPUT — Only after all checks, give the final answer.

By forcing the AI to follow this sequence — one step at a time — we can catch mistakes before the final output.

The Code

import 'dotenv/config';
import { OpenAI } from 'openai';

const client = new OpenAI();

async function main() {
  const SYSTEM_PROMPT = `
    You are an AI assistant who works on START, THINK, EVALUATE, OUTPUT format.
    Always break down problems, evaluate correctness, and only give the final output after all thinking steps are done.

    Output JSON format:
    { "step": "START | THINK | EVALUATE | OUTPUT", "content": "string" }
  `;

  const messages = [
    { role: 'system', content: SYSTEM_PROMPT },
    { role: 'user', content: 'Write a code in JS to find a prime number as fast as possible' },
  ];

  while (true) {
    const response = await client.chat.completions.create({
      model: 'gpt-4.1-mini',
      messages,
    });

    const rawContent = response.choices[0].message.content;
    const parsed = JSON.parse(rawContent);

    messages.push({ role: 'assistant', content: JSON.stringify(parsed) });

    if (parsed.step === 'START') {
      console.log(`🔥`, parsed.content);
      continue;
    }

    if (parsed.step === 'THINK') {
      console.log(`\t🧠`, parsed.content);

      messages.push({
        role: 'developer',
        content: JSON.stringify({
          step: 'EVALUATE',
          content: 'Nice, you are going on the correct path',
        }),
      });

      continue;
    }

    if (parsed.step === 'OUTPUT') {
      console.log(`🤖`, parsed.content);
      break;
    }
  }

  console.log('Done...');
}

main();

source: https://github.com/piyushgarg-dev/genai-js-1.0

Why This Works

• No blind guessing — AI must show its reasoning.

• Error catching — Mistakes are caught in EVALUATE before they reach the user.

• Composable — You can swap the evaluator with another AI (LLM-as-a-judge) or a human reviewer.

• Transparent — Every decision step is visible for debugging.

Beyond Math

This technique isn’t just for calculations.
You can apply it to:

• Debugging code

• Medical diagnosis

• Legal reasoning

• Complex business workflows

Any time reasoning matters more than speed, this approach turns your LLM into a thinking partner instead of a pattern-matcher.

If you ask Zomato’s chatbot to write a Python script, it’s not going to start coding for you. Why?
Because it’s only working within the scope it has been assigned.

That “scope” is set through something called a system prompt - the AI’s hidden set of instructions that define its purpose, tone, and boundaries.

What is a System Prompt?

A system prompt is like the AI’s job description. It tells the AI:

• Who it should be (you can define a name, tone, personality, style)

• What it should and shouldn’t do

• How it should answer (structure of output)

If AI is a chef, the system prompt is the recipe card you hand them before they start cooking. Everything after that follows those instructions.

Example:

• Without a system prompt: AI gives a neutral, general answer.

• With a system prompt: AI answers exactly as instructed, e.g., “Explain in the style of a service manager.”

Why System Prompts Matter

1. Scope Control – Keeps the AI focused on its purpose (like Zomato bot sticking to food queries).

2. Consistency – Ensures the same tone and style across responses.

3. Role Setting – Makes AI behave like a teacher, coder, marketer, or even a poet.

4. User Experience – Gives a unique personality to the interaction.

Without a well-designed system prompt, the AI can feel generic or confused.

Types of Prompting

Now that you know what a system prompt is, let’s look at the types of prompting you can use when interacting with AI models like GPT.

1. Zero-Shot Prompting

The model is given a direct question or task without any prior example.

• Example:
  "Translate 'I am learning AI' into French."

When to use: The task is simple and widely understood by AI (common cases)

2. Few-Shot Prompting

You give a few examples before asking the main question. (Around 100-150 examples is a good range)

• Example:

  English: Hello → French: Bonjour
  English: Thank you → French: Merci
  English: How are yo....(more examples)

When to use: To get a specific style, tone, or format.

3. Chain-of-Thought Prompting

The model is encouraged to break down a problem into multiple small sub-problems, and evaluate each one by one, reasoning each step before giving the final output.

• Example:
  "Explain your reasoning before solving: 27 × 14."

When to use: For reasoning-heavy tasks like math, logic, or planning.

4. Self-Consistency Prompting

Think of this like asking multiple friends the same question and then going with the answer most of them agree on.

In AI’s case, instead of generating just one chain of thought, it generates multiple reasoning paths and then picks the answer that comes up the most.

AI Process:

• Reasoning Path 1 → Answer: 405

• Reasoning Path 2 → Answer: 405

• Reasoning Path 3 → Answer: 402

Final Answer: 405 (picked because it appeared most often).

When to use:

• High-stakes reasoning tasks (math, planning, legal analysis).

• When you need more reliability and less chance of a random mistake.

How it works:
It’s like cross-checking your homework before submission, either by your own or by any other friend or person - different “thoughts” compete, and the most consistent one wins.

5. Persona Prompting

This is when you tell the AI to pretend to be a specific person, profession, or character so it answers from that perspective.

It’s like asking your friend, “Imagine you’re a chef, how would you make Maggi?” — their answer will change based on the role they take.

Example:
Instruction: You are an experienced financial advisor. Explain the basics of budgeting to a college student.

AI Output:
"Alright, first thing you need to do is track your expenses... Think of your income as a pizza and your budget as how you slice it."

When to use:

• Customer support (AI acts like a polite support rep).

• Education (AI acts like a history teacher or coding tutor).

• Creative writing (AI acts like Shakespeare or a movie director).

How it works:
It sets the context and tone before the AI even sees your question, so responses feel more natural and aligned with that persona.

Just like Zomato’s chatbot won’t write code for you, AI systems will only perform as well as the instructions they’re given.
The system prompt is the hidden boss that defines those instructions.

Pair it with the right prompting technique — zero-shot, few-shot, chain-of-thought, or role-based — and you can make AI work exactly the way you want.

The next time you talk to an AI, remember: the magic starts before you type.

Now imagine you have a special parrot.
Not just any parrot - this one doesn’t only repeat words you say.
This parrot has read millions of books, heard endless stories, and seen countless conversations.
So when you ask it something, instead of just repeating, it thinks for a moment and gives you a brand-new, meaningful answer.

Why is GPT called GPT?

GPT stands for Generative Pretrained Transformer:

• Generative → It can generate new sentences, stories, or answers.

• Pretrained → Before you even talk to it, it has already learned from a huge amount of text from books, articles, and the internet.

• Transformer → A special type of computer model that understands patterns in text and figures out what should come next.

How it Works (Parrot Version)

Think of GPT like that intelligent parrot:

1. It listens to your question → “What’s the capital of India?”

2. Remembers all the reading it has done → "Oh! I’ve read this many times in books and articles."

3. Speaks back in its own words → “The capital of India is New Delhi.”

Why It Feels Magical

The magic is that GPT doesn’t just repeat facts.
You can ask it to tell a bedtime story, solve a riddle, write a poem, or explain maths - and it will do it instantly like a Disney movie parrot who never forgets anything it learned.

So, GPT is like that clever parrot in our neighborhood who not only repeats what it hears but also learns so much that it can talk about new things you never taught it directly. The only difference? GPT doesn’t need food, water, or a cage - it just needs data and some good training. Next time you chat with GPT, think of it as a super-parrot that’s read the whole world’s books, newspapers, and websites… and is now ready to chat with you about anything from “why the sky is blue” to “how to make a paper rocket.”

When a computer works with text, it can’t directly understand sentences the way we understand.
It needs to break the text into smaller pieces so it can process them step-by-step.

Those smaller pieces are called tokens.
Basically, tokenization is the process of splitting text into tokens.

What Do I Mean?

Example in Plain English
Think of a sentence:

I love samosas

When we do Tokenization, we could break it into:

Let’s say we break it based on Word-level tokens:

["I", "love", "samosas"]

Now, Character-level tokens:

["I", " ", "l", "o", "v", "e", " ", "s", "a", "m", "o", "s", "a", "s"]

Generally, in Machine Learning & AI, the tokenizer converts an input into a unique number assigned to that exact word. And you know that computers are better with numbers, this also eliminates the confusion that can occur when someone misspells or miscases the input.
For example:

[
  46530,
  4,
  55530,
  4,
  82663
]

Note: I used my recently made tool Tea Tokenizer here.
Link: https://teatokenizer.monc.space

Why is it Important?

Tokenization is like splitting a long message into smaller parts so the computer can read it one step at a time.

• Without tokenization, the computer sees the entire sentence as one giant block of text and can’t figure out where words or parts of words start and end.

• With tokenization, the text becomes small chunks (tokens) that the computer can store, search, and process efficiently.

In Short, Tokenization is cutting big text into small, meaningful chunks so a computer can handle it.

So, I decided to take on a challenge: Could I explain vector embeddings to my mom without using a single technical jargon?

At first, I did some searches, some LLM help, they were guiding me to use the example of mangoes and fruits, but recently, when my sister was getting ready for school, and was yelling, “Where’s my uniform?”

My mom replied:

“Kitni baar boli hoon ki uniform upar wale shelf me hai!”
(How many times have I told you, the uniform is on the top shelf!)

Then I thought I could explain this concept with this trending topic (like we express ourselves using viral Instagram Memes)

Explanation: The Wardrobe Example

In our Indian home, clothes are never just thrown in. Mom has a system: My & my sibling’s wardrobe is in my sibling's room, but behind the separate doors. Mom’s wardrobe is in her room.

Inside each wardrobe, there is a left door and a Right door. Behind each door, there are 3 sections: Top shelf uniforms/professional outfits, Middle shelf for T-shirts/Topwears, Bottom shelf pants/jeans/bottomwears, and Innerwear hung on hooks inside the doors.

It’s neat, predictable, and easy to find things.

Mapping: Turning Clothes into Coordinates

Let’s say I want my blue T-shirt, and I tell my mother in the kitchen; Mom, where’s my blue T-shirt. After that, the usual dialogue “Saare kaam main hi karoon… +200 more lines”, she will tell me the exact place of my t-shirt, without even going to the room.

Now, for you (tech people), we can describe it as:
[Room: My room, Door: Left, Shelf: Middle, Type: T-shirt]

If we turn that into numbers:

• My room -> 0

• Left door -> 0

• Middle shelf -> 1

Now the position of my T-shirt is: [0, 0, 1, T-shirt]

Similarly:

• My sibling’s T-shirt: [0, 1, 1, T-shirt]

• Mom’s T-shirt: [1, 0, 1, T-shirt]

These numbers are like coordinates on a map, telling us exactly where something lives in our “wardrobe space.”

Wardrobe To Vectors

Now Imagine This…

Instead of clothes, what if we’re arranging words, sentences, images, or sounds, basically data/information?

In AI, vector embeddings store words, sentences, images, or sounds in a multi-dimensional space (different rooms or wardrobes) where:

Similar meanings are stored close together (like my T-shirt and my sibling’s T-shirt, and corresponding to similar coordinate (same shelf position) distance in different dimensions (wardrobe).

Different meanings are far apart (like my T-shirt and a cooking pan)

Example: All my books, notebooks, and stationery are placed in the nearby places in my room, but my bike key is hanging in the living room.

Why Vector Embeddings Matter

By storing meanings as coordinates, AI can:

• Find similar things (search “T-shirt” and get all T-shirts)

• Group related items (keep all uniforms together)

• Understand relationships (knowing my and my sibling’s T-shirts are similar kinds of items)

This is why embeddings are used not only in AI, but way before, already being used in search engines, chatbots, recommendation systems & more.

How You Can Explain It Too

1. Pick a familiar system - wardrobes, library bookshelves, kitchen spice rack, recommend picking the recent topic in your house, your mother, or whoever you are going to explain had just discussed or encountered. Also, break things into sections (dimensions).

2. Although I didn’t explain this part to my mother but, you can show how each item’s location can be described as numbers, as explained in the section “Mapping: Turning Clothes into Coordinates.”

3. Connect the example to how AI stores meanings & highlight how “closeness” in this space means similarity.

Conclusion

Just like Mom knows exactly where my jeans are without opening every shelf, AI knows where “mango” is and which other words are sitting right next to it.

Message from my Mom

“Thanks for reading this article, and I know you’re surely gonna forget tomorrow where your favourite jeans are, but don’t forget to like, and share your thoughts or anything I have missed. Follow me to get more articles like this.”