Introduction to
Big Data Analytics

Data = raw facts & figures collected from various sources to be analyzed for insights. Computer data may be text, images, audio, software programs.

Big Data is a massive, continuously growing collection of data so huge and complex that no typical data management technology can effectively store or process it. Data in Petabytes (10¹⁵ bytes) qualifies. ~90% of today's data was generated in the last 3 years.

Effective scaling methods: Database sharding, memory caching, moving to cloud, and separating read-only and write-active databases. Combine all for next-level scaling.

Trending Technologies: Hadoop Ecosystem, Apache Spark, NoSQL Databases, R Software, Predictive Analytics, Prescriptive Analytics.

Collection of technologies, frameworks, and tools used to handle, process, analyze, and derive insights from large datasets.

NoSQL = "Not Only SQL". High-performance agile processing of large-scale information. Inherently unstructured, distributed architecture, horizontal scaling.

Open-source framework for storing and processing large datasets (GBs to PBs) on clusters of commodity hardware. Developed at Apache Software Foundation. In 2008, defeated supercomputers for sorting terabytes.

Data Locality Concept: Computational logic is sent to cluster nodes containing data, not the other way around.

Core processing component providing the logic of processing. Uses divide-and-conquer on cluster nodes.

-- Word Count Example --
Input:
  "Welcome to Hadoop Class"
  "Hadoop is good"
  "Hadoop is bad"

Map Output (key-value pairs):
  (Welcome,1), (to,1), (Hadoop,1), (Class,1)
  (Hadoop,1), (is,1), (good,1)
  (Hadoop,1), (is,1), (bad,1)

Shuffle & Sort → Group by key

Reduce Output:
  bad → 1  |  Class → 1  |  good → 1
  Hadoop → 3  |  is → 2  |  to → 1  |  Welcome → 1

Introduced in Hadoop 2.x. Separates resource management from processing layer. Allows multiple data processing engines (Spark, Flink, etc.) to run on same cluster.

Platform built on Hadoop for analyzing large datasets. Developed by Yahoo (2006). Uses Pig Latin (SQL-like procedural language). Pig Engine converts scripts into MapReduce jobs.

Pig Data Model: Atom (single value/field) → Tuple (ordered fields, like a row) → Bag (unordered set of tuples, like a table) → Map (key-value pairs) → Relation (bag of tuples)

Data warehouse tool on top of Hadoop. Developed at Facebook → Apache. Uses HiveQL (HQL) which converts to MapReduce jobs internally. Stores schema in MetaStore, data in HDFS.

Open-source distributed NoSQL database built on HDFS. Modeled after Google's BigTable. Provides real-time random read/write to large datasets. Column-oriented storage. Written in Java.

Distributed coordination service for managing large sets of hosts. Services: Naming Service, Configuration Management, Cluster Management, Leader Election, Locking & Synchronization, Highly Reliable Data Registry.

Architecture: Client → Server → Ensemble (min 3 nodes) → Leader (auto-elected) → Follower. ZooKeeper Data Model: znode tree structure with path separators (/).

Lightning-fast cluster computing. Built on top of Hadoop MapReduce. 100x faster than Hadoop for in-memory computations. Written in Scala. Supports Python, Java, Scala, R, SQL APIs.

RDD (Resilient Distributed Dataset): Fault tolerant, Distributed, Dataset (partitioned collection). Immutable. Lazy evaluation. Two operations: Transformations (create new RDD) and Actions (pass results to driver).

Free, open-source, distributed wide-column store NoSQL database. Designed for large data across commodity servers with high availability and no single point of failure. Written in Java. Developed by Apache (originally at Facebook for Inbox Search, 2008).

Origin: Blend of Google BigTable (data model + storage) + Amazon Dynamo (distribution features). Facebook open-sourced in 2008 → Apache Incubator 2009 → Top-level Apache project 2010.

Eric Brewer (2000): A distributed system can only guarantee 2 of 3 properties simultaneously:

Cassandra follows AP (Availability + Partition Tolerance) with BASE consistency (Basically Available Soft State Eventual Consistency).

-- Create Keyspace (namespace for tables) --
CREATE KEYSPACE Employees
  WITH REPLICATION = {
    'class': 'SimpleStrategy',
    'replication_factor': 3
  };

-- Use Keyspace --
USE Employees;

-- Create Table --
CREATE TABLE example_table (
  id UUID PRIMARY KEY,
  name TEXT,
  age INT,
  tags LIST<TEXT>,
  categories SET<TEXT>,
  attributes MAP<TEXT, TEXT>
);

-- Insert (BATCH for multiple) --
BEGIN BATCH
  INSERT INTO example_table (id, name, age) VALUES (uuid(), 'Alice', 30);
  INSERT INTO example_table (id, name, age) VALUES (uuid(), 'Bob', 25);
APPLY BATCH;

-- Read / Update / Delete --
SELECT * FROM example_table WHERE id = some_uuid;
UPDATE example_table SET age = 31 WHERE id = some_uuid;
DELETE FROM example_table WHERE id = some_uuid;
TRUNCATE example_table; -- delete all rows

-- Counter Example --
CREATE TABLE page_views (article_id UUID PRIMARY KEY, views counter);
UPDATE page_views SET views = views + 1 WHERE article_id = ?;

-- TTL: auto-expire in 30 seconds --
INSERT INTO userlogin (userid, password) VALUES (1, 'pass') USING TTL 30;

-- Export / Import CSV --
COPY mykeyspace.mytable TO '/path/output.csv' WITH HEADER = true;
COPY mykeyspace.mytable FROM '/path/input.csv' WITH HEADER = true;

Open-source document-oriented database. Categorized under NoSQL. Developed by MongoDB.Inc under SSPL license (Feb 2009). Data stored in BSON (Binary JSON) format. Max document size: 16MB. Max nesting depth: 100 levels.

// Database Operations
use mydb                          // create/switch db
show dbs                         // list databases
db.dropDatabase()                 // drop current db
db.createCollection("students")   // create collection
show collections                  // list collections

// CREATE
db.records.insertOne({ name: "Alice", age: 25 });
db.records.insertMany([{ name: "Bob" }, { name: "Carol" }]);

// READ
db.records.find()                          // all docs
db.records.find({ name: "Alice" })         // filter
db.records.findOne({ age: 25 })            // first match

// UPDATE
db.records.updateOne({ name: "Alice" }, { $set: { age: 26 } });
db.records.updateMany({ age: 25 }, { $set: { status: "active" } });
db.records.replaceOne({ name: "Alice" }, { name: "Alice", age: 30 });

// DELETE
db.records.deleteOne({ name: "Alice" });
db.records.deleteMany({ age: { $lt: 18 } });

Data warehousing tool. Processes structured data in Hadoop. Uses HDFS for storage, MapReduce for execution, RDBMS for metadata. Developed at Facebook → Apache Hive.

-- DDL: Create --
CREATE DATABASE IF NOT EXISTS employee_db;
CREATE TABLE IF NOT EXISTS employee (
  emp_id INT, emp_name STRING, salary FLOAT
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
  STORED AS TEXTFILE;

-- DDL: Show, Describe, Alter, Drop --
SHOW DATABASES; SHOW TABLES; DESCRIBE employee;
ALTER TABLE employee ADD COLUMNS (age INT);
ALTER TABLE employee RENAME TO staff;
DROP TABLE IF EXISTS employee;
TRUNCATE TABLE employee;  -- removes rows, keeps structure

-- DML: Load, Insert, Select --
LOAD DATA LOCAL INPATH '/home/user/emp.csv' INTO TABLE employee;
INSERT INTO TABLE employee VALUES (1, 'John', 50000);
SELECT emp_name, salary FROM employee WHERE salary > 40000;

-- Export / Import --
EXPORT TABLE employee TO '/user/hive/export/employee';
IMPORT TABLE employee_copy FROM '/user/hive/export/employee';

-- Create partitioned table --
CREATE TABLE sales (id INT, amount FLOAT)
PARTITIONED BY (year INT, month INT)
STORED AS TEXTFILE;

-- Static partition load --
LOAD DATA INPATH '/data/sales_2025.txt'
INTO TABLE sales PARTITION (year=2025, month=10);

-- Dynamic partition insert --
SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
INSERT INTO TABLE sales PARTITION (year, month)
SELECT id, amount, year, month FROM temp_sales;

High-level abstraction from MapReduce. Developed by Yahoo (2006). Pig Engine converts Pig Latin scripts into MapReduce jobs. Two execution environments: local JVM (small data) and Hadoop cluster (distributed).

-- Load data --
A = LOAD '/pigdemo/student.tsv' AS (rollno:int, name:chararray, gpa:float);

-- FILTER: select rows where GPA > 4 --
B = FILTER A BY gpa > 4.0;

-- FOREACH: transform each row --
C = FOREACH A GENERATE UPPER(name);

-- LIMIT: first 3 rows --
D = LIMIT A 3;

-- GROUP: group by field --
G = GROUP A BY rollno;

-- DISTINCT: unique values --
U = DISTINCT A;

-- ORDERBY: sort --
O = ORDER A BY gpa DESC;

-- JOIN: join two relations --
J = JOIN A BY rollno, B BY id;

-- DUMP or STORE output --
DUMP B;
STORE B INTO '/output/result';

In-memory cluster computing. Built on Hadoop MapReduce. 100x faster for large-scale in-memory processing. 10x faster for disk-based processing. Written in Scala (U.C. Berkeley). Supports Python, Java, Scala, R, SQL.

Stream Data = continuous, unbounded flow of data generated over time. Unlike static databases, stream data is dynamic, constantly evolving, potentially infinite. Examples: IoT sensors, financial transactions, social media, GPS.

• High Velocity: Elements arrive rapidly. Missing real-time window = data loss. Archival access less desirable.
• Main Memory Execution: Algorithms must run in RAM. Secondary storage access must be infrequent.
• Multiplicity of Streams: Many simultaneous streams overwhelm available memory.
• Memory Constraints: Limited memory makes problems challenging → need approximate algorithms.

-- Continuous monitoring query on sensor stream --
SELECT * FROM SensorDataStream
WHERE Temperature > 28 AND Humidity > 70;

-- Sample stream data --
Timestamp             SensorID  Temperature  Humidity
2024-03-11 12:00:01   Sensor1   25.5°C       60%
2024-03-11 12:00:03   Sensor2   22.0°C       55%
2024-03-11 12:00:05   Sensor3   28.3°C       70%   ← triggers alert!

Data Science is a multidisciplinary field to address challenges in Big Data. Involves gathering, analyzing, and decision-making from vast amounts of data. Finds patterns and makes future predictions. Applies to ALL data — big and small.

• Mathematics & Applied Mathematics
• Applied Statistics / Data Analysis
• Solid Programming (R, Python, Julia, SQL)
• Data Mining & Machine Learning
• Database Storage and Management