WM-BigQuery-Demo: Enterprise Data Warehouse Architecture Simulation
Enterprise Google BigQuery demonstration project simulating Microsoft Fabric, Azure, Snowflake, and Atlas free-tier architectural workflows by Christopher Lazok.
Special Waste Operations Hub: Data & Optimization Pipeline
📖 Project Overview
This repository serves as the Agile project log and DevOps documentation for the Special Waste Operations Data Optimization proof-of-concept. The goal of this project is to demonstrate a highly scalable, governed, and optimized data pipeline for inbound logistics and waste manifest processing.
Architecture Stack:
Storage & Compute: Google BigQuery (Serverless Data Warehouse)
Data Governance & Lineage: OpenMetadata (Atlan alternative)
Presentation / UI Layer: Next.js, Tailwind CSS, Vercel (RayFin Architecture)
DevOps & Tracking: GitHub Projects
🎯 Agile Project Log (Epics & Features)
📦 EPIC 1: Cloud Storage & Ingestion Strategy
Objective: Establish a secure, high-performance, and serverless data storage layer for raw EPA logistics manifests without incurring idle compute costs.
Feature 1.1: Environment Provisioning
Task: Evaluate Microsoft Fabric, Snowflake, and Google Cloud BigQuery for cost-to-performance ratio on static flat-file ingestion.
Task: Provision GCP Sandbox Environment (WM-Logistics-Demo).
Decision Log: Opted for BigQuery over Dataproc/Spark to eliminate cluster management overhead and leverage serverless SQL for CSV workloads.
Feature 1.2: Data Ingestion
Task: Create BigQuery Dataset (special_waste_ops).
Task: Perform native UI bulk upload of manifests.csv (Simulating Bronze/Silver layer ingestion).
Task: Execute schema auto-detection to map data types (FLOAT, VARCHAR, INT).
🛡️ EPIC 2: Data Governance & Lineage (The Atlan Requirement)
Objective: Implement enterprise-grade data cataloging and automated metadata tracking to ensure data trustworthiness for the Business Optimization team.
Feature 2.1: Governance Engine Deployment
Task: Deploy OpenMetadata Sandbox (utilizing OMAG standards equivalent to Atlan).
Feature 2.2: Automated Lineage Crawling
Task: Configure GCP IAM Service Account (metadata-crawler) with BigQuery Data Viewer and Metadata Viewer roles.
Task: Generate secure JSON credentials for OpenMetadata API ingestion.
Task: Execute metadata crawl to automatically map table relationships, column definitions, and data ownership tags.
📊 EPIC 3: Presentation & Executive UI (Front-End)
Objective: Decouple the presentation layer from the data plumbing to provide a zero-latency, highly accessible executive dashboard.
Feature 3.1: UI Prototyping
Task: Utilize v0.dev AI generation to rapidly prototype a Next.js/React layout.
Task: Implement Waste Management brand guidelines (Deep Greens, clean Whites).
Feature 3.2: Metric Definition (KPIs)
Task: Define mock data hooks for 4 core operational metrics: Total Manifests, Avg Freight Cost/Mile, Total Airspace Volume (Tons), and AI Flagged Volume Anomalies.
Feature 3.3: CI/CD Deployment
Task: Commit finalized React code to GitHub main branch.
Task: Link repository to Vercel for automated CI/CD static builds and global edge-network hosting.
🛠️ Data Dictionary (Governance Mapping)
Field Name
Data Type
Business Definition
Governance Status
Manifest_ID
VARCHAR
Unique alphanumeric key for inbound waste shipments.
✅ Verified (Ops)
Facility_Name
VARCHAR
Target disposal site destination name.
✅ Verified (Compliance)
Raw_Tons
FLOAT
Original certified scale weight recorded at entry point.
✅ Verified (Scale Ops)
Carrier_Cost
FLOAT
Negotiated contract transportation rate per mile traveled.
⚠️ Audited (Procurement)
Optimized_Vol
FLOAT
Calculated field adjusted by validation engine.
🤖 AI-Enriched
Anomaly_Flag
VARCHAR
Automated flag indicating data saturation risks.
🤖 AI-Enriched
Maintained by: Christopher Lazok | Business Analyst II# WM-BigQuery-Demo Demo of Google BigQuery Features Epics - Simulates MS FABRIC Azure Snowflake Atlas, utilizing Fee Tier Solutions