Job Description
Job description
Responsibilities
- Lead the architecture design and implementation of advanced analytics solutions using Azure Databricks Fabric The ideal candidate will have a deep understanding of big data technologies data engineering and cloud computing with a strong focus on Azure Databricks along with Strong SQL
- Work closely with business stakeholders and other IT teams to understand requirements and deliver effective solutions
- Oversee the endtoend implementation of data solutions ensuring alignment with business requirements and best practices
- Lead the development of data pipelines and ETL processes using Azure Databricks PySpark and other relevant tools
- Integrate Azure Databricks with other Azure services eg Azure Data Lake Azure Synapse Azure Data Factory and onpremise systems
- Provide technical leadership and mentorship to the data engineering team fostering a culture of continuous learning and improvement
- Ensure proper documentation of architecture processes and data flows while ensuring compliance with security and governance standards
- Ensure best practices are followed in terms of code quality data security and scalability
- Stay updated with the latest developments in Databricks and associated technologies to drive innovation
Essential Skills
- Strong experience with Azure Databricks including cluster management notebook development and Delta Lake
- Proficiency in big data technologies eg Hadoop Spark and data processing frameworks eg PySpark
- Deep understanding of Azure services like Azure Data Lake Azure Synapse and Azure Data Factory
- Experience with ETLELT processes data warehousing and building data lakes
- Strong SQL skills and familiarity with NoSQL databases
- Experience with CICD pipelines and version control systems like Git
- Knowledge of cloud security best practices
Soft Skills
- Excellent communication skills with the ability to explain complex technical concepts to nontechnical stakeholders
- Strong problemsolving skills and a proactive approach to identifying and resolving issues
- Leadership skills with the ability to manage and mentor a team of data engineers
Nice to have Skills
- Power BI for dashboarding and reporting
- Microsoft Fabric for analytics and integration tasks
- Spark Streaming for processing realtime data streams
- Familiarity with Azure Resource Manager ARM templates for infrastructure as code IaC practices
Experience
- Demonstrated expertise of 8 years in developing data ingestion and transformation pipelines using DatabricksSynapse notebooks and Azure Data Factory
- Solid understanding and handson experience with Delta tables Delta Lake and Azure Data Lake Storage Gen2
- Experience in efficiently using Auto Loader and Delta Live tables for seamless data ingestion and transformation
- Proficiency in building and optimizing query layers using Databricks SQL
- Demonstrated experience integrating Databricks with Azure Synapse ADLS Gen2 and Power BI for endtoend analytics solutions
- Prior experience in developing optimizing and deploying Power BI reports
- Familiarity with modern CICD practices especially in the context of Databricks and cloudnative solutions
Skills
Mandatory Skills : Azure Cloud Architecture, Cloud Architecture, Cloud Solution Architecture, Enterprise Cloud Architecture
Required Skills : Azure
Basic Qualification :
Additional Skills :
This is a high PRIORITY requisition. This is a PROACTIVE requisition
Background Check : No
Drug Screen : No
Job Tags