Gonzalo
Cruz.

Overview

End-to-end data pipeline processing ~1.08M rows of European TripAdvisor restaurant data. The core constraint was memory: the dataset can't fit in RAM, so every stage processes data in 50k-row chunks from first to last.

Architecture

• 5-stage Airflow DAG using TaskFlow API: Extract → Clean → EDA → Preprocess → Stream
• IncrementalPCA and StandardScaler fitted via partial_fit — online learning, no full dataset in memory
• OHE vs label encoding chosen per column based on cardinality thresholds
• Kafka producer with acks=all and sub-batch delivery for fault tolerance
• All parameters (chunk size, thresholds, Kafka config) centralized in a single config.toml

Key challenge

Making scikit-learn transformers work incrementally. The standard fit/transform API requires the full dataset. Switching to partial_fit required careful state management — the fitted transformers had to persist across chunk iterations and be reused for the transform pass.

Overview

Distributed real-time trending analysis using PySpark DStreams. Connects to a live TCP tweet feed, filters for US-only content, and maintains a 5-minute sliding window with 10-second slide intervals to rank hashtags continuously.

The interesting part

The naive approach — recomputing the full window on each slide — is O(window size). Instead, the pipeline uses inverse reduction functions (invFunc): on each slide, only new arrivals are added and expired tweets are subtracted. This makes each update O(slide interval), not O(window size). A 5-minute window with 10-second slides is 30x cheaper per update.

Stack decisions

• DStreams over Structured Streaming — lower-level, but invFunc is not available in the Structured API
• TCP socket source simulates a real-time feed for local testing

Overview

AFINN-111 lexicon sentiment scoring on a 314MB Twitter JSON dataset using MapReduce on a fully Dockerized Hadoop cluster. The goal was to build and operate the infrastructure from scratch, not just run a managed service.

Infrastructure

• Multi-node Hadoop cluster (NameNode + DataNodes) set up with Docker Compose and custom images
• HDFS for distributed storage, YARN for resource management and job scheduling
• Shell-scripted HDFS file upload and MapReduce job submission — fully automated deploy

MapReduce design

Mapper tokenizes tweet text and scores each token against the AFINN lexicon. Reducer aggregates per-tweet scores. The Dockerized setup means the entire cluster can be reproduced with a single docker compose up.

Overview

Binary classification of stroke risk on a heavily imbalanced medical dataset (5110 patients, ~5% positive). The core decision was to optimize for Recall over Accuracy — in a clinical setting, a missed stroke costs far more than a false alarm.

Pipeline

• SMOTE for class imbalance — synthetic oversampling of the minority class
• RFECV for feature selection: 20+ features reduced to 12 most predictive
• Decision threshold tuned post-training to maximize Recall on validation
• Models compared: Logistic Regression, MLP, LinearSVC, RBF-SVM

Results

Logistic Regression won with 0.88 Recall and 0.84 AUC on the validation set. RBF-SVM achieved high accuracy but failed on Recall — it was predicting the majority class almost exclusively. Diagnosing that failure mode was the most instructive part of the project.

Overview

Churn prediction on 10,000 bank customers with a research component: Naive Bayes, AdaBoost, and K-NN implemented from mathematical first principles in base R, then benchmarked against library implementations.

The from-scratch experiment

The hand-rolled Naive Bayes outperformed the library version on Recall — 34.55% vs 23.24%. The reason: the library applies Laplace smoothing by default, which dampens probabilities for rare classes. Implementing it from scratch meant I could tune this directly for the imbalanced dataset.

Full pipeline

• PCA for dimensionality reduction before boosted models
• K-Means for customer segmentation — reveals behavioral clusters before classification
• XGBoost as final model, best overall performance after segmentation

Overview

Regression study predicting burned area in Montesinho Natural Park from meteorological and fire index covariates. The dataset has severe zero-inflation (many fires burn near-zero area) and extreme right skew — OLS is the wrong tool from the start.

Model progression

• OLS baseline — exposed heteroscedasticity and zero-inflation problems
• Gamma GLM — handles positive skew and non-constant variance
• Inverse Gaussian GLM — heavier tail, better fit for extreme values
• GAM with smoothing splines via mgcv — final model, best fit

Key finding

GAMs revealed non-linear threshold effects for temperature and Drought Code that were invisible to OLS and GLMs — above certain thresholds, burned area increases sharply. This is a physical phenomenon the parametric models couldn't capture. Model choice here is a scientific decision, not a default.

Overview

Dual-mode robotic vehicle built on an STM32 Nucleo-L152RE. Operates autonomously using ultrasonic obstacle detection, or manually via Bluetooth commands from an Android app. Firmware written in C using STM32 HAL.

Hardware

• STM32 Nucleo-L152RE microcontroller
• HC-SR04 ultrasonic sensor — three distance thresholds: forward (>20cm), decelerate (10–20cm), hard stop (<10cm)
• L298N H-Bridge driver + two DC motors
• HC-05 Bluetooth module for RC mode (Forward, Backward, Left, Right, Stop)
• Piezoelectric buzzer for audio warning on hard stop

Firmware architecture

Fully interrupt-driven: TIM2 captures ultrasonic echo pulses, TIM3 generates PWM for motor speed, USART2 handles async Bluetooth commands, GPIO manages direction. Soft braking — progressive PWM duty cycle reduction rather than abrupt stop — protects the drivetrain from mechanical stress.

Overview

Implementation of Whitehead's (1983) unified theory for sample size determination in clinical trials. The goal: find the minimum sample size to detect a deviation from a baseline event probability of p₀ = 0.003 with Type I error α = 0.025 and 80% power.

Methods

• Mathematical derivation of Fisher Information and Score Statistics for binomial distributions
• Whitehead's formula applied to get the theoretical sample size
• Parametric tests: Chi-Square and Exact Binomial for validation
• Wilcoxon Rank-Sum Test as a non-parametric alternative when normality assumptions break down
• Monte Carlo simulation: binomial draws used to compare theoretical vs empirical Type I error rates

Key finding

At p = 0.003, asymptotic approximations (Score Test) are computationally efficient but require sufficiently large n to be valid — small probabilities push the asymptotics hard. The Exact Binomial test provides a ground truth check, and the Wilcoxon confirms robustness when the normality assumption is questionable.

Overview

Analysis and forecasting of four different time series using SARIMA models. The workflow for each series follows the Box-Jenkins methodology: identify, estimate, and diagnose — repeated until the residuals behave like white noise.

Methodology

• Stationarity testing with ADF and KPSS — both tests used together since they have opposite null hypotheses
• Time series decomposition into trend, seasonal, and remainder components
• ACF and PACF plots to determine candidate AR, MA, and seasonal orders
• Model selection via AIC and BIC across a grid of SARIMA(p,d,q)(P,D,Q)s candidates
• Residual diagnostics: Ljung-Box test, ACF of residuals, Q-Q plots to validate white noise assumption

What makes it interesting

Working with four structurally different series forces you to make different modeling decisions for each one — different differencing orders, different seasonal periods, different levels of heteroscedasticity. The project is as much about diagnosing what's wrong with a model as it is about fitting one.