What are the responsibilities and job description for the Big Data Engineer position at Vertogic?
Job Description
Hi
Currently looking for Big Data Engineer with one of our direct client. Please see the job details below and let me know if you would be interested in this role. If interested, please send me a copy of your resume, your contact details, your availability and a good time to connect with you.
Location- Costa Mesa, CA
Job Title- Big Data Engineer
Key Responsibilities:
- Perform large-scale data processing using Python and Apache Spark.
- Utilize AWS services like S3, EMR, EC2 for data storage and processing.
- Schedule, monitor, and troubleshoot data pipelines with Apache Airflow.
- Ensure the data quality, observability, and integrity across all our data infrastructure.
- Implement data structures and algorithms to optimize data processing and pipelines.
- Write and maintain robust Python test cases to validate data processing tasks.
- Oversee data versioning, backups, and restoration to ensure data resilience and availability.
- Collaborate with cross-functional teams to implement best practices in data management.
Skills & Qualifications:
- Proven experience in a Data Engineering role, with a focus on large-scale data processing.
- Strong knowledge of Python programming language.
- Proficiency with AWS services, including S3, EMR, and EC2.
- Hands-on experience with PySpark and Apache Airflow.
- Good understanding and hand on experience on Data process optimizations.
- Excellent understanding of data structures and algorithms.
- Experience in writing Python test cases to validate data processing tasks.
- Deep understanding of data quality assurance, data versioning, backups, and restoration.
- Familiarity with Git or Bitbucket for version control.
Sr. Network Voice Engineer
MDS (Micro-Data Systems) -
Orange, CA
Big Data/NoSQL/JAVA Consultant
Sonsoft Inc -
Irvine, CA
Big data Architect ( Experience 12 Years)
EROS Technologies Inc. -
Costa Mesa, CA