Project description:

Leveraging data & software engineering best practices we want to build a reliable architecture for data processing and sharing and build valuable data products?. The focus of our efforts lies in building a streaming pipeline to ingest, process and distribute usage data.

Service description:

Supporting the maintenance of an AWS pipeline Supporting the migration to the new data platform (Databricks)
Implementing change requests after discussing the scope and effort with the responsible product owner
Supporting incidents

Requirements:

Python
Spark
Terraform
In depth Databricks experience (eg working with sensitive data in production, deeper references than POC level)
AWS
Practical experience in various applications (not just POC level) required everywhere!