Data Engineer Databricks/Dataiku

Responsibilities

• Design, develop and maintain scalable data pipelines on the Databricks platform
• Manage data ingestion processes from various sources into the Databricks environment.
• Implement and optimize ETL (Extract, Transform, Load) processes to ensure efficient data transformation and integration.
• Ensure data quality, integrity, and security across all data pipelines.
• Monitor and troubleshoot data pipeline performance, implementing necessary maintenance and updates.
• Document data pipeline processes, technical specifications and best practices.
• Realize knowledge transfer towards the Data engineer team which will take over the responsibility to run the platform in the future
• You demonstrate and use your developer experience to develop and maintain complex data transformation steps using Dataiku under specific guidelines (dev vs automation node, use of Dataiku’s low code recipes, etc.)