Personal Blog

Hi, I'm Sourav.
I write about things
that matter to me.

Code, ideas, and whatever I'm figuring out. No noise.

All Posts

  • Deploying Spark to Production

    The finale of the series — cluster managers compared (Standalone, YARN, Kubernetes), client vs. cluster deploy modes, key spark-submit flags, and a checklist for running Spark for real.

    spark big-data tutorial pyspark kubernetes devops

  • Structured Streaming: Real-Time Data with Spark

    How Spark's micro-batch streaming engine works, reading from sockets and Kafka, output modes, windowed aggregations over event time, and handling late data with watermarks.

    spark big-data tutorial pyspark streaming

  • Performance Tuning: Partitioning, Caching, and Shuffles

    A practical tuning checklist — when and how to cache, fixing data skew, choosing partition counts, and reading executor memory metrics before they turn into out-of-memory failures.

    spark big-data tutorial pyspark performance

  • Jobs, Stages, and Tasks: How Spark Executes a Query

    What actually happens between calling an action and getting a result — the DAG scheduler, how shuffles split work into stages, and how to read the Spark UI like an execution trace.

    spark big-data tutorial pyspark performance

  • Joins, Aggregations, and Window Functions

    Join strategies Spark chooses between (and how to influence that choice), grouped aggregations, and window functions for running totals, rankings, and row-relative comparisons.

    spark big-data tutorial pyspark spark-sql

  • Reading and Writing Data: Files, Formats, and Databases

    CSV, JSON, Parquet, and JDBC in Spark — how to read and write each, why file format choice matters more than people expect, and partitioning your output sensibly.

    spark big-data tutorial pyspark parquet

  • DataFrames and Spark SQL

    Meet the API you'll actually use day to day — DataFrames, schemas, the Catalyst optimizer, and how to mix DataFrame code with plain SQL in the same job.

    spark big-data tutorial pyspark spark-sql

  • RDDs: The Original Building Block

    A proper deep dive into Resilient Distributed Datasets — transformations vs. actions, narrow vs. wide dependencies, lazy evaluation, and why RDDs still matter even though most code today uses DataFrames.

    spark big-data tutorial pyspark rdd

  • Installing Spark and Your First Job

    Get PySpark running locally, launch a Spark shell, run the classic word count job, and take a tour of the Spark UI to see what actually happened.

    spark big-data tutorial pyspark

  • Apache Spark 101: What It Is and Why It Exists

    Kicking off a 10-part Spark tutorial series — what problem Spark solves, how it compares to Hadoop MapReduce, and the core architecture you need in your head before writing a line of code.

    spark big-data tutorial pyspark

  • Artemis: Going Back to the Moon — This Time to Stay

    NASA's Artemis program aims to return humans to the Moon for the first time in over 50 years, and build the foundation for a journey to Mars.

    artemis nasa moon future

  • Project Apollo: One Giant Leap

    The story of the program that landed humans on the Moon — from tragedy to triumph, from Earth to the Sea of Tranquility.

    apollo nasa moon space-history

  • Project Gemini: Learning to Fly Before You Can Land

    Between Mercury and Apollo, ten Gemini missions answered the hardest questions about surviving a journey to the Moon.

    gemini nasa space-history

  • Project Mercury: America's First Step Into Space

    How seven test pilots became the Original Seven, and America raced to prove humans could survive in space.

    mercury nasa space-history