📢 Notice 📢

Introduction to Data Mining

July 22, 2025 2 minute read

Data mining bridges data and knowledge, combining cleaning, analysis, and modeling to deliver actionable insights in real-world applications.

What is Data Mining?

Data mining (DM) is the process of discovering insightful, novel, and understandable patterns or models from large-scale data.

Key points:

We mine knowledge, not raw data.
Closely related to Machine Learning (ML), AI, and Statistics:
- AI: Simulates human intelligence
- ML: Learns patterns from data
- DM: Focuses on discovering knowledge from large datasets
Applications:
- Market Basket Analysis (items frequently bought together)
- Fault detection from sensor signals
- Trend discovery in event logs
Challenges:
- Data issues: Noise, missing values, corruption
- Knowledge issues: Limited domain expertise, ground truth
- Computing: Large-scale, distributed data, limited memory/power
Typical Process:
1. Data Collection
2. Data Preprocessing (cleaning, feature extraction, selection)
3. Analytical Modeling (train, test, validate, deploy, evaluate)

Order of attributes/instances does not matter:

Quantitative Multi-dimensional Data: Continuous numerical features (e.g., age, temperature).
Categorical or Mixed Data: Discrete labels, sometimes combined with numerical features.
Binary / Set Data: Yes/No or 0/1 attributes, common in transactions for association mining.

Order or relationships do matter:

Time-Series Data: Sequential observations over time (e.g., temperature logs).
- Contextual Attributes: Timestamps that provide temporal context
- Behavioural Attributes: Measured values that change over time (e.g., temperature)
- Multiple Attributes: Includes several behavioural variables recorded concurrently
- Example: (temperature, humidity, vibration) recorded per second
Discrete Sequences / Strings: Ordered symbolic data like event logs or DNA sequences. These are conceptually similar to time-series data, but with a key distinction:
- Contextual Attribute: Timestamp or position within the sequence
- Behavioural Attribute: Categorical value (e.g., event type, nucleotide)
- Key Difference: Categorical values instead of continuous numerical ones
Spatial Data: Positional or movement data using coordinates (e.g., object tracking).
Network / Graph Data: Entities represented as nodes, with relationships as edges (e.g., social networks, citation graphs).

Clustering (Descriptive): Group similar data (e.g., customer segmentation).
Classification (Predictive): Predict class labels for new instances.
Regression (Predictive): Predict continuous values (e.g., price prediction).
Outlier Detection: Identify unusual patterns (e.g., fraud detection).
Association Rule Mining (Descriptive): Discover items frequently co-occurring (e.g., Market Basket Analysis).

Supervised Learning: Labeled data
- Examples: Decision Trees, k-NN, SVM, Naive Bayes, Linear Regression
Unsupervised Learning: No labels, discover intrinsic structures
- Examples: K-means, Hierarchical Clustering
Association Rule Mining: Apriori, FP-Growth, Eclat
Outlier Detection Methods: Extreme value analysis, Histograms, Local Outlier Factor, Classification-based

Data mining is widely applied across domains:

Insurance Fraud Detection – Identifying suspicious claims
Netflix Prize – Improving movie recommendations via hidden pattern discovery
Structural Health Monitoring – Outlier detection for abnormal signals