VIP20
本课程属于：006-Data Science and Engineering Analytics English Technical Courses · 共 20 门相关课程，组合学习更完整。
查看所属VIP全部课程 →
← 返回所属目录 咨询 / 购买

Engineering Analytics Platform Design and Data Pipelines

30 Chapters
View Course Samples Online
01
Engineering analytics platform overview
core capabilities, stakeholders, data maturity models, and platform architecture layers.
analyticsplatform
02
Requirements engineering for analytics platforms
use case discovery, SLAs, data contracts, and non-functional requirements.
Requirementsanalytics
03
Data source integration
batch and streaming connectors, change data capture, API ingestion, and file-based ingestion patterns.
integrationstreaming
04
Ingestion pipeline design
idempotency, exactly-once semantics, backpressure handling, and dead-letter queues.
Ingestionpipeline
05
Data lake and lakehouse architecture
storage formats, partitioning strategies, table formats, and catalog integration.
lakehousearchitecture
06
Batch processing frameworks
Apache Spark, Flink batch mode, and workflow orchestration with Airflow or Dagster.
processingframeworks
07
Stream processing pipelines
Kafka, Flink, windowing, event time vs processing time, and stateful operations.
processingpipelines
08
Data transformation and modeling
dbt, SQL-based transformations, dimensional modeling, and slowly changing dimensions.
transformationmodeling
09
Data quality engineering
validation frameworks, anomaly detection, data observability, and SLA monitoring.
validationframeworks
10
Metadata management and data catalog
technical metadata, business glossary, lineage tracking, and discovery.
Metadatamanagement
11
Data governance and security
access control, column-level security, encryption, masking, and compliance.
governancesecurity
12
Orchestration and scheduling
DAG design, dependency management, retries, alerting, and backfill strategies.
Orchestrationscheduling
13
Infrastructure as code for analytics platforms
Terraform, Kubernetes operators, and environment promotion.
Infrastructureanalytics
14
Compute and storage optimization
cost governance, autoscaling, spot instances, and storage tiering.
optimizationgovernance
15
Data pipeline testing
unit tests, integration tests, contract tests, and production data validation.
pipelineintegration
16
CI/CD for data pipelines
version control, artifact management, deployment automation, and rollback.
CI/CDpipelines
17
Monitoring and observability
pipeline metrics, logging, distributed tracing, and incident response.
Monitoringobservability
18
Real-time analytics serving
OLAP engines, caching layers, materialized views, and query acceleration.
Real-timeanalytics
19
Feature store design
offline/online consistency, feature pipelines, and integration with ML workflows.
offline/onlineconsistency
20
Data mesh and decentralized ownership
domain-oriented pipelines, data products, and federated governance.
decentralizedownership
21
Multi-tenant analytics platform design
isolation, resource quotas, and tenant-specific pipeline configurations.
Multi-tenantanalytics
22
Data pipeline migration
legacy ETL to modern ELT, incremental migration patterns, and dual-run validation.
pipelinemigration
23
Performance tuning for pipelines
shuffle optimization, skew handling, partitioning, and file compaction.
Performancepipelines
24
Reliability engineering
fault tolerance, checkpointing, idempotent reprocessing, and disaster recovery.
Reliabilitytolerance
25
Cost and resource management
chargeback models, query optimization, and pipeline efficiency metrics.
resourcemanagement
26
Compliance and privacy in pipelines
GDPR, CCPA, PII detection, consent management, and data retention.
Compliancepipelines
27
Platform extensibility
plugin architectures, custom operators, SDKs, and self-service onboarding.
Platformextensibility
28
End-to-end case study
designing a batch and streaming pipeline for a real-world analytics use case.
End-to-enddesigning
29
End-to-end case study
building a governed, observable, and cost-efficient analytics platform.
End-to-endbuilding
30
Future directions
AI-assisted pipeline development, serverless analytics, and open table format evolution.
directionsAI-assisted