Showing posts with label Indexing. Show all posts
Showing posts with label Indexing. Show all posts

MongoDB Queries Tutorial: Filters and Projections Explained


Finding Documents with Queries: Filters & Projections

A Magical Treasure Hunt in MongoDB : For beginners to Expert Level

Quick Overview: This MongoDB tutorial explains how to find documents using filters and projections. Whether you’re a beginner or an expert, you’ll learn step-by-step query examples using mongosh and MongoDB Compass to search, filter, and display data efficiently.


๐Ÿ“˜ Table of Contents


Imagine you’re in the Hero Academy Library, and you need to find one special book from thousands. You can ask:

  • “Show me all fire heroes.”
  • “Give me only their names and powers.”

In MongoDB, this is called querying! You use filters (what to find) and projections (what to show). This tutorial is a fun treasure hunt, super easy for a student, but packed with pro-level secrets for experts.

We’ll use:

  • Our Hero Academy from before
  • mongosh (command line)
  • MongoDB Compass (click & search)
  • Real images


Part 1: What Are Filters & Projections?

TermReal-Life ExampleMongoDB Meaning
Filter“Find all red toys”Conditions to select documents
Projection“Show only toy name and color”Fields to show or hide

Fun Fact: Filter = “Who to invite?”
Projection = “What info to print on the card?”



Part 2: Sample Data (Let’s Load Our Heroes!)


use heroAcademy
db.heroes.insertMany([
  { name: "Aarav", power: "Super Speed", level: 5, isActive: true, team: "Alpha", skills: ["run", "jump"] },
  { name: "Priya", power: "Invisibility", level: 7, isActive: true, team: "Alpha", skills: ["hide", "sneak"] },
  { name: "Rohan", power: "Fire Control", level: 4, isActive: false, team: "Beta", skills: ["flame", "shield"] },
  { name: "Sanya", power: "Telekinesis", level: 6, isActive: true, team: "Beta", skills: ["lift", "fly"] },
  { name: "Karan", power: "Ice Blast", level: 3, isActive: true, team: "Alpha", skills: ["freeze", "snow"] }
])


Part 3: Basic Filters – “Find Who?”

1. Find All Heroes

db.heroes.find()

→ Shows everything

2. Find Active Heroes

db.heroes.find({ isActive: true })

→ Only heroes with isActive: true
Like: “Show me only students who came to school today.”

3. Find by Exact Match

db.heroes.find({ power: "Fire Control" })

→ Only Rohan

4. Find by Number

db.heroes.find({ level: { $gt: 5 } })

→ Priya (7) and Sanya (6)

Operators You’ll Love:

OperatorMeaningExample
$gtGreater thanlevel: { $gt: 5 }
$ltLess thanlevel: { $lt: 4 }
$gteGreater or equallevel: { $gte: 6 }
$lteLess or equallevel: { $lte: 5 }
$neNot equalteam: { $ne: "Alpha" }
$inIn a listname: { $in: ["Aarav", "Priya"] }

5. Find in Arrays

db.heroes.find({ skills: "fly" })

→ Sanya (has “fly” in skills)

db.heroes.find({ skills: { $all: ["run", "jump"] } })

→ Only Aarav

6. Find in Nested Fields

db.heroes.find({ "team": "Alpha" })


Part 4: Projections: “Show Only What I Want”

By default, .find() shows all fields. Use projection to pick!

Syntax:

db.collection.find(filter, projection)

1. Show Only Name and Power


db.heroes.find(
  { isActive: true },
  { name: 1, power: 1, _id: 0 }
)

{ "name": "Aarav", "power": "Super Speed" }
{ "name": "Priya", "power": "Invisibility" }

Rules:
1 = Include this field
0 = Hide this field
Never mix 1 and 0 (except for _id)
Always hide _id with _id: 0 if not needed

2. Hide Skills


db.heroes.find(
  { team: "Alpha" },
  { skills: 0 }
)

→ Shows all except skills



Part 5: Combine Filter + Projection


db.heroes.find(
  { isActive: true, team: "Alpha" },
  { name: 1, level: 1, _id: 0 }
)

{ "name": "Aarav", "level": 5 }
{ "name": "Priya", "level": 7 }
{ "name": "Karan", "level": 3 }

Beginner Win: Like making a custom report card!



Part 6: Using Compass : Click to Query!

  1. Open Compass → heroAcademy → heroes
  2. Use Filter Bar

{ "isActive": true, "team": "Alpha" }

Compass Filter

  1. Use Projection

{ "name": 1, "level": 1, "_id": 0 }

Click & See! No typing needed.



Part 7: Advanced Filters (Pro Level)

1. Text Search (Need index!)


db.heroes.createIndex({ name: "text" })
db.heroes.find({ $text: { $search: "Priya" } })

Important: $text queries only work on fields that have a text index. Each collection can have just one text index, so make sure no other text index exists before creating a new one. You can check existing indexes with db.heroes.getIndexes().

2. Regex (Pattern Match)


db.heroes.find({ name: /^A/ })        // Starts with A
db.heroes.find({ name: /an$/i })      // Ends with "an", case-insensitive

Note: The patterns /^A/ and /an$/i are JavaScript regular expressions (regex). MongoDB supports the same regex syntax used in JavaScript, so you can easily search by text patterns like “starts with”, “ends with”, or “contains”.

3. Logical Operators


// AND (default)
db.heroes.find({ level: { $gt: 5 }, isActive: true })

// OR
db.heroes.find({
  $or: [
    { power: "Fire Control" },
    { power: "Ice Blast" }
  ]
})

// NOT
db.heroes.find({ team: { $ne: "Alpha" } })

4. Array Queries


// Has exactly 2 skills
db.heroes.find({ skills: { $size: 2 } })

// Has skill AND level > 5
db.heroes.find({
  skills: "fly",
  level: { $gt: 5 }
})

5. Dot Notation (Nested)


// If hero had address:
db.heroes.find({ "address.city": "Delhi" })


Part 8: Mini Project - Hero Search Engine!

1. Find Top Heroes (level ≥ 7)


db.heroes.find(
  { level: { $gte: 7 } },
  { name: 1, power: 1, level: 1, _id: 0 }
).pretty()

2. Find Inactive Heroes


db.heroes.find(
  { isActive: false },
  { name: 1, team: 1, _id: 0 }
)

3. Find Alpha Team Flyers


db.heroes.find(
  { team: "Alpha", skills: "fly" },
  { name: 1, _id: 0 }
)

Real-World Connection: The same query patterns you used for your Hero Academy can power real-world applications too. For example, in an e-commerce site, you might filter products by category and price, or in a school app, find students by grade and attendance. MongoDB queries make these kinds of searches fast and flexible!



Part 9: Pro Tips for All Levels

For Students & Beginners

  • Use Compass filter bar – just type!
  • Start with { field: value }
  • Always use .pretty() in mongosh

For Medium Learners


db.heroes.find()
         .sort({ level: -1 })   // High to low
         .limit(3)              // Top 3

Count matches:


db.heroes.countDocuments({ team: "Alpha" })

For Experts


db.heroes.aggregate([
  { $match: { isActive: true } },
  { $group: { _id: "$team", count: { $sum: 1 } } }
])

Index for speed:


db.heroes.createIndex({ level: 1, team: 1 })

Use explain() to debug:


db.heroes.find({ level: 5 }).explain("executionStats")


Part 10: Cheat Sheet (Print & Stick!)

Query TypeExample Code
Find alldb.heroes.find()
Filterdb.heroes.find({ level: 5 })
With operatordb.heroes.find({ level: { $gt: 5 } })
Projectiondb.heroes.find({}, { name: 1, _id: 0 })
Filter + Projectiondb.heroes.find({ team: "Alpha" }, { name: 1 })
Pretty print.pretty()
Count.countDocuments({})


Part 11: Common Mistakes & Fixes

MistakeFix
Forgetting quotesUse "power": "Fire" not power: Fire
Wrong field nameCheck with findOne()
Using == instead of :Use : in JSON: { level: 5 }
Forgetting _id: 0Add it to hide ID


Final Words

You’re a Query Master!
You just:
Used filters to find exact heroes
Used projections to show only what you want
Hunted in shell and Compass
Learned pro tricks like $or, indexing, aggregation

Your Mission:


db.heroes.find(
  { "skills": "run" },
  { name: 1, power: 1, _id: 0 }
).pretty()

Who did you find?
You’re now a Certified MongoDB Detective!



Resources:



Bonus Tip for Experts ๐Ÿง 

You can even compare values between fields within the same document using the $expr operator, a powerful feature for complex logic.


db.heroes.find({
  $expr: { $gt: ["$level", 5] }
})

This finds heroes whose level is greater than 5 without needing a fixed number in your filter!

Keep hunting treasures in your data!

Indexing and Query Optimization with SQL: A Fun Beginner Tutorial (Part 1)

Indexing and Query Optimization with SQL: A Simple and Fun Tutorial for Everyone-Part 1

Welcome back to our magical SQL adventure! In our previous tutorials, we learned how to manage data, join tables, use transactions, and even create triggers in our toy store database. Now, let’s explore something that makes your database super fast: indexing and query optimization. These are like giving your database a treasure map to find data quickly and a shortcut to work smarter. This tutorial is designed to be easy and useful for beginners and experienced users, and covers key aspects of indexing and query optimization. We’ll use SQLite and Python with our toystore.db database, keeping it fun and simple like organizing a toy shop with a magic speed boost!


What are Indexing and Query Optimization?

Imagine you’re looking for your favorite toy in a huge toy store. Without a guide, you’d have to check every shelf, which takes forever! An index is like a map or a table of contents that tells the database exactly where to find data, making searches lightning-fast. Query optimization is like finding the shortest path to get that toy, ensuring your SQL commands (queries) run quickly and efficiently.

In our toy store, we’ll:

  • Create an index to make searches faster.
  • Learn how to write smart queries that use indexes.
  • Explore tips to optimize queries for speed.

Why Learn Indexing and Query Optimization?

Indexing and query optimization are awesome because:

  • They’re Fast: They make your database find data in a snap.
  • They’re Simple: Indexes are easy to create, and optimization is like organizing your work.
  • They’re Useful: Fast databases are critical for apps, websites, and games.
  • They’re Fun: It’s like giving your database super speed powers!
  • They Build on SQL: If you know SELECT or WHERE from our earlier tutorials, you’re ready for this.

Let’s dive into our toy store and make our database zoom!


Getting Started

We’ll use Python with SQLite to run our SQL commands, just like before. Make sure you have Python installed (download it from python.org if needed). SQLite comes with Python, so no extra setup is required. You can also use DB Browser for SQLite to see your data visually, but we’ll focus on Python code for clarity.

We’ll work with our toystore.db database, using the Toys table and a Sales table. To make things interesting, we’ll add more data to simulate a bigger store. Here’s the setup code:

import sqlite3

# Connect to the database
conn = sqlite3.connect('toystore.db')
cursor = conn.cursor()

# Create Toys table
cursor.execute('''
    CREATE TABLE IF NOT EXISTS Toys (
        ToyID INTEGER PRIMARY KEY,
        Name TEXT,
        Type TEXT,
        Price REAL,
        Stock INTEGER
    )
''')

# Create Sales table
cursor.execute('''
    CREATE TABLE IF NOT EXISTS Sales (
        SaleID INTEGER PRIMARY KEY,
        ToyID INTEGER,
        Quantity INTEGER,
        TotalPrice REAL,
        SaleDate TEXT,
        FOREIGN KEY (ToyID) REFERENCES Toys(ToyID)
    )
''')

# Clear existing data
cursor.execute("DELETE FROM Toys")
cursor.execute("DELETE FROM Sales")

# Add toys
toys = [
    ('Robot', 'Action Figure', 30.00, 10),
    ('Jigsaw', 'Puzzle', 10.00, 15),
    ('Teddy', 'Stuffed Animal', 15.00, 8),
    ('Car', 'Model', 20.00, 12),
    ('Doll', 'Doll', 25.00, 5)
]
cursor.executemany("INSERT INTO Toys (Name, Type, Price, Stock) VALUES (?, ?, ?, ?)", toys)

# Add sales
sales = [
    (1, 2, 60.00, '2025-09-01'),
    (2, 3, 15.00, '2025-09-01'),
    (3, 1, 30.00, '2025-09-02'),
    (4, 5, 25.00, '2025-09-02'),
    (1, 4, 20.00, '2025-09-03')
]
cursor.executemany("INSERT INTO Sales (ToyID, Quantity, TotalPrice, SaleDate) VALUES (?, ?, ?, ?)", sales)

conn.commit()
conn.close()
print("Toy store database ready for indexing and optimization!")

What’s Happening?

  • Toys table: Stores toy details with ToyID, Name, Type, Price, and Stock.
  • Sales table: Tracks sales with SaleID, ToyID, Quantity, TotalPrice, and SaleDate.
  • We added 5 toys and 5 sales to make our database busy.
  • cursor.executemany: Inserts multiple rows efficiently.

Our Toys table looks like this:

ToyIDNameTypePriceStock
1RobotAction Figure30.0010
2JigsawPuzzle10.0015
3TeddyStuffed Animal15.008
4CarModel20.0012
5DollDoll25.005

What is an Index?

An index is like a shortcut that helps the database find data faster. Without an index, SQLite checks every row in a table (called a full table scan), which is slow for big tables. An index is like a phone book that lists names and their locations, so you can jump straight to the right spot.

For example, if you often search for toys by Name, an index on Name makes those searches faster.

Creating an Index

Syntax:

CREATE INDEX index_name ON table_name (column);

Example:

import sqlite3

conn = sqlite3.connect('toystore.db')
cursor = conn.cursor()

# Create index on Name
cursor.execute("CREATE INDEX IF NOT EXISTS idx_toy_name ON Toys (Name)")

conn.commit()
print("Index on Toy Name created!")

# Test a query using the index
cursor.execute("SELECT * FROM Toys WHERE Name = 'Robot'")
print("Found Robot:", cursor.fetchone())

conn.close()

What’s Happening?

  • CREATE INDEX idx_toy_name ON Toys (Name): Creates an index named idx_toy_name on the Name column.
  • IF NOT EXISTS: Avoids errors if the index already exists.
  • The index makes searches like WHERE Name = 'Robot' faster.

Output:

Index on Toy Name created!
Found Robot: (1, 'Robot', 'Action Figure', 30.0, 10)

This shows that the query quickly found the toy named "Robot" using the index we created. Without an index, SQLite would have had to scan every row.


What is Query Optimization?

Query optimization means writing SQL commands that run as fast as possible. Indexes help, but you also need to write smart queries. Here are key tips:

  1. Use Specific Columns: Select only the columns you need, not *.
  2. Use Indexes: Search on indexed columns.
  3. Avoid Unnecessary Data: Filter early with WHERE.
  4. Use Joins Wisely: Ensure joins use indexed columns.

Example: Optimizing a Query

→ Ready to supercharge your queries? Click here for Part 2!


Database Performance Tuning: Tips & Techniques for SQL and NoSQL Optimization

๐Ÿ”ท Part 13: Database Performance Tuning – Optimize Your Queries and System


๐Ÿ“ Introduction

A well-designed database is essential, but even the best database can slow down over time due to inefficient queries, growing data, or poor configuration. Performance tuning helps your database respond faster and handle more users without crashing.

This part introduces fundamental concepts and practical tips for tuning both SQL and NoSQL databases.You can also read Part 12: Database Backup & Recovery if you missed it.


๐Ÿ”ธ 1. Identify Performance Bottlenecks

  • Use EXPLAIN or EXPLAIN PLAN to analyze query execution.

  • Monitor slow queries and their impact.

  • Check system resources: CPU, memory, disk I/O.


๐Ÿ”น 2. Indexing Strategies

  • Create indexes on columns frequently used in WHERE, JOIN, or ORDER BY clauses.

  • Avoid over-indexing; too many indexes slow down writes.

  • Consider composite indexes for queries filtering on multiple columns.



๐Ÿงช Try It Yourself – Analyze a Query

Use the EXPLAIN statement to analyze how your database executes a query. This helps identify slow joins or missing indexes.

EXPLAIN SELECT * FROM users WHERE last_login < NOW() - INTERVAL 7 DAY;

๐Ÿ”— Learn more about EXPLAIN in MySQL


๐Ÿ”ธ 3. Query Optimization

  • Select only necessary columns, avoid SELECT *.

  • Use appropriate joins and avoid nested queries if possible.

  • Filter early — apply WHERE clauses to reduce rows processed.


๐Ÿ”น 4. Database Configuration

  • Tune cache sizes, connection pools, and memory allocation.

  • Adjust parameters like max_connections, work_mem (PostgreSQL), or innodb_buffer_pool_size (MySQL).

  • Enable query caching where applicable.


๐Ÿ”ธ 5. NoSQL Specific Tips

  • Design schema based on query patterns.

  • Use denormalization to reduce expensive joins.

  • Optimize shard keys and partitions for distributed systems.


๐Ÿ“ Summary

Tuning Aspect SQL NoSQL
Bottleneck Tools EXPLAIN, slow query logs, monitoring Profiling, explain plans (e.g., MongoDB explain)
Indexing Single & composite indexes Index on query fields, secondary indexes
Query Optimization Select columns, joins, filters Schema design, query patterns
Configuration Cache, memory, connections Cache, sharding, replication

๐Ÿš€ Advanced Tuning Tips (For Power Users)

  • Enable query profiling tools like pg_stat_statements (PostgreSQL) or Performance Schema (MySQL).
  • Use Redis or Memcached as a caching layer to reduce read load.
  • Explore parallel query execution (PostgreSQL 10+ or MySQL 8.0+).
  • Benchmark performance with tools like sysbench or Apache JMeter.

❓ Frequently Asked Questions (FAQ)

What is database performance tuning?

Database performance tuning is the process of optimizing database queries, indexes, configuration settings, and resources to improve response time and scalability.

How do I identify slow queries in MySQL?

You can use the EXPLAIN statement, enable the slow query log, or use performance monitoring tools like MySQL Performance Schema or pt-query-digest.

What are composite indexes, and when should I use them?

Composite indexes combine multiple columns into a single index. They're useful when your query filters on two or more columns in a specific order.

Do NoSQL databases support query optimization?

Yes. While the techniques differ, NoSQL databases like MongoDB support query profiling, indexing, and schema design optimization for performance tuning.

What tools can I use for performance benchmarking?

Common tools include sysbench, Apache JMeter, pgbench for PostgreSQL, and native monitoring dashboards from your DBMS.



๐Ÿ’ฌ What’s Next?

In Part 14, we’ll dive into Data Modeling Best Practices — designing efficient and scalable database schemas.


Have questions about tuning or optimization tips of your own? Leave a comment and share your experience below! Let’s learn from each other. ๐Ÿ‘‡

๐Ÿ‘‰ Want more like this? Follow or bookmark the blog for weekly hands-on Python and DB tutorials.

๐Ÿ“ข Found this useful? Share it with others who want to tune their databases!


Hands-On SQL & MongoDB Indexing Tutorial | Query Optimization Exercise

๐Ÿงช Hands-On Exercise: Indexing and Query Optimization


๐ŸŽฏ Objective:

Understand the impact of indexes by comparing query performance with and without indexing in both SQL and MongoDB.


๐Ÿ”น Scenario:

You're working with a database of 10,000 students, and you need to search for a student by name.


A. SQL Exercise (MySQL or PostgreSQL)


๐Ÿ“Œ Step 1: Create and Populate the Table

CREATE TABLE Students (
  StudentID INT PRIMARY KEY,
  Name VARCHAR(100),
  Class VARCHAR(10),
  Email VARCHAR(100)
);

-- Insert sample data (loop or script to add 10,000+ rows for better realism)
-- Example:
INSERT INTO Students (StudentID, Name, Class, Email)
VALUES
(1, 'Aisha P', '10A', 'aisha@example.com'),
(2, 'Ravi Patel', '10B', 'ravi@example.com'),
-- Add thousands more...
(10000, 'Simran Joshi', '12B', 'simran@example.com');

๐Ÿ“Œ Step 2: Run Query Without Index

EXPLAIN ANALYZE
SELECT * FROM Students WHERE Name = 'Simran Joshi';

๐Ÿ“‹ Note: Record the execution time or rows examined.


๐Ÿ“Œ Step 3: Add an Index

CREATE INDEX idx_name ON Students(Name);

๐Ÿ“Œ Step 4: Run the Same Query Again

EXPLAIN ANALYZE
SELECT * FROM Students WHERE Name = 'Simran Joshi';

Compare performance: Look for reduced cost, fewer rows scanned, and faster time.



๐Ÿ” Example Output (PostgreSQL with Index):

Index Scan using idx_name on students  (cost=0.28..8.30 rows=1 width=52)
  Index Cond: (name = 'Simran Joshi')
Planning Time: 0.080 ms
Execution Time: 0.102 ms

๐Ÿ†š Compare this with the earlier output (without index), which might look like:

Seq Scan on students  (cost=0.00..194.00 rows=1 width=52)
  Filter: (name = 'Simran Joshi')
Planning Time: 0.095 ms
Execution Time: 5.223 ms


B. MongoDB Exercise


๐Ÿ“Œ Step 1: Insert Sample Data

for (let i = 1; i <= 10000; i++) {
  db.students.insertOne({
    student_id: i,
    name: `Student${i}`,
    class: `10${i % 5}`,
    email: `student${i}@example.com`
  });
}

// Insert the student to query:
db.students.insertOne({
  student_id: 10001,
  name: "Simran Joshi",
  class: "12B",
  email: "simran@example.com"
});

๐Ÿ“Œ Step 2: Run Query Without Index

db.students.find({ name: "Simran Joshi" }).explain("executionStats");

๐Ÿ“‹ Note the totalDocsExamined and executionTimeMillis.


๐Ÿ“Œ Step 3: Add an Index on Name

db.students.createIndex({ name: 1 });

๐Ÿ“Œ Step 4: Re-run the Query

db.students.find({ name: "Simran Joshi" }).explain("executionStats");

✅ You should see:

  • Lower totalDocsExamined

  • Faster executionTimeMillis

  • WinningPlan using IXSCAN instead of COLLSCAN



๐Ÿ” Example Output (MongoDB after indexing):

{
  "executionStats": {
    "executionTimeMillis": 1,
    "totalDocsExamined": 1,
    "nReturned": 1
  },
  "queryPlanner": {
    "winningPlan": {
      "stage": "IXSCAN"
    }
  }
}

๐Ÿ†š Before indexing, the output might show:

{
  "executionStats": {
    "executionTimeMillis": 15,
    "totalDocsExamined": 10001,
    "nReturned": 1
  },
  "queryPlanner": {
    "winningPlan": {
      "stage": "COLLSCAN"
    }
  }
}

๐Ÿง  Reflection Questions

  1. How did indexing affect query speed in SQL and MongoDB?

  2. What would happen if you indexed every column? Why is that a bad idea?

  3. Which type of queries benefit most from indexing?


๐Ÿ’ก Try online:


๐ŸŽ“ Now You Completed this lab? Great! You're ready to continue with :

  • Part 9: SQL Joins vs NoSQL Referencing


Featured Post

MongoDB Update & Delete Made Easy: Step-by-Step for Beginners

Updating and Deleting Data in MongoDB A Magical Eraser & Pencil Adventure - For Beginner to Expert Level Imagine you have a super-...

Popular Posts