Why Index Everything? Get Smarter with Selective Indexing in MongoDB

Backend
1

I recently encountered a use case that made me rethink how I was applying indexes in MongoDB.
You know how we usually just slap an index on an entire field and move on? Turns out, sometimes you only want that index to apply to certain documents — not the whole collection.

Let’s unpack that.

What are indexes (and why they matter)

Think of an index like the index section in a book.
Instead of flipping through every page to find “Marketing,” you glance at the front and see:

Marketing → pages 34, 120, 305

In databases, an index helps MongoDB jump directly to the documents it needs instead of scanning everything.

  • Regular index – improves read performance on a field.
  • Unique index – ensures no two documents share the same value for that field (like unique email IDs).
  • Compound index – combines multiple fields to optimize queries that filter on both, e.g. { category: 1, publishDate: -1 }.

A quick Mongo example:

db.blogs.createIndex({ category: 1, publishDate: -1 })

Now queries filtering by both fields are lightning fast.

:gear: Under the Hood — How Indexes Work and Choosing the Right Type

It’s worth understanding what’s happening behind the scenes.
MongoDB’s indexes are built on B-tree–like data structures, which maintain keys in sorted order for fast lookups and range queries.
Each index entry stores the indexed field value(s) and a pointer to the corresponding document in the collection.

There are also other index types available:

  • B-tree (default) → great for equality and range queries.
  • Hashed indexes → distribute values across buckets for quick equality lookups (but not for ranges).
  • Wildcard indexes → dynamically index multiple fields or subdocuments.
  • Text indexes → optimized for full-text search.
  • Geospatial indexes → for coordinates and location queries.

While indexes significantly improve read performance, they also add overhead to writes.
Every time you insert, update, or delete a document, MongoDB must also update all relevant indexes — which introduces write latency and increases storage usage.

So when designing indexes, it’s not just about enforcing business logic , but also about aligning with business performance requirements:

  • If your workload is read-heavy, investing in multiple, well-tuned indexes makes sense.
  • If it’s write-heavy, you’ll want to minimize indexes to reduce latency.
  • For balanced workloads, partial or compound indexes can give a middle ground — performance without excess cost.

In short: choose indexes strategically, not just logically.

The twist — uniqueness only for some documents

Here’s the situation.
We have a collection where each document has:

{ configName, status }

status can be "Published", "Draft", or "Deleted".
We want to ensure that no two documents share the same configName when their status is "Published" or "Draft".
However, when a config is "Deleted", we keep it for audit/history — and we don’t care if duplicates exist there.

At first, this seems fine with a unique compound index:

db.configs.createIndex({ configName: 1, status: 1 }, { unique: true })

And it does work… until we hit this case:

{ configName: "X", status: "Published" }
{ configName: "X", status: "Deleted" }

So far, all good — the (configName, status) pairs are unique.
But then we delete the published one (i.e. change its status from "Published" → "Deleted").
Now there are two documents with { configName: "X", status: "Deleted" }.

MongoDB throws a duplicate key error, because the unique index is being violated.
That’s the exact pain point: we want the uniqueness guarantee only for certain statuses — “Published” and “Draft” — not “Deleted”.

The fix — MongoDB partial indexes

Enter partial indexes.
They allow you to index (and optionally enforce uniqueness) only for documents that meet a given filter condition.

From MongoDB’s official docs:

“Partial indexes only index the documents in a collection that meet a specified filter expression.”

“If you specify both partialFilterExpression and unique: true, the uniqueness constraint only applies to the documents that meet the filter.”

— MongoDB Documentation

Here’s how you solve our use case:

db.configs.createIndex(
  { configName: 1, status: 1 },
  {
    unique: true,
    partialFilterExpression: { status: { $in: ["Published", "Draft"] } }
  }
)

This index only applies to documents where the status is "Published" or "Draft".
Among those, MongoDB enforces uniqueness on (configName, status).
Documents with status = "Deleted" aren’t included — and won’t cause conflicts.

A few caveats

  • You can’t use negative operators ($ne, $nin) in the partialFilterExpression.
    So you can’t write { status: { $ne: "Deleted" } }.
    You must explicitly list what to include:
    { status: { $in: ["Published", "Draft"] } }
  • If new statuses get added (like "Archived") and you want the uniqueness rule to apply there too, you’ll need to drop and recreate the index with the updated filter.
  • Partial indexes can’t be used with _id, shard keys, or combined with sparse.
  • Queries need to include the same condition (or a subset of it) for MongoDB to actually use the partial index effectively.

Why this is awesome

  • Business alignment → only “active” configs must be unique; “Deleted” ones can duplicate.
  • Performance win → smaller index, faster writes, less storage.
  • Clarity → the database enforces the business rule directly, reducing logic clutter in application code.

TL;DR

Indexes speed things up.
Partial indexes make them smarter.

If you ever find yourself thinking “I want this index to apply only to some rows/some values of the field”, MongoDB’s partial indexes are the tool you’re looking for — especially when combined with unique: true.
Just remember: explicitly include what you need indexed, and manage updates carefully as your data model evolves.

:speech_balloon: The index management, dropping and recreating on change in index is still a pain point
If anyone has more insights or better patterns for managing selective indexing, drop them in the comments — let’s discuss!

:writing_hand: Proofread Credits: Swasthik

5 Likes