Apache Spark Specialist

Location Singapore
Discipline
Job Reference AppSpark-IBM_1563362261
Salary Negotiable
Consultant Email james.andalan@manpower.com.sg
EA License No. 02C3423

Job Description

The Client's Analytics team is looking for a strong technologist to help implement Spark/Scala jobs. The developer must not only know the Spark API well: They must also know about the pitfalls of distributed storage, how to structure a data processing pipeline that has to handle the 5V of Big Data-volume, velocity, variety, veracity, and value-and how to turn that into maintainable code.

Responsibilities:

  • Help write analytics code, services and components in Java, Apache Spark, Kafka, Storm, Redis and related technologies such as Hadoop, Zookeeper
  • Responsible for systems analysis - Design, Coding, Unit Testing and other SDLC activities
  • Requirement gathering and understanding, Analyse and convert functional requirements into concrete technical tasks and able to provide reasonable effort estimates
  • Spark query tuning and performance optimisation
  • Work proactively, independently and with global teams to address project requirements, and articulate issues/challenges with enough lead time to address project delivery risks
  • Providing expertise in technical analysis and solving technical issues during project delivery
  • Code reviews, test case reviews and ensure code developed meets the requirements

Qualifications:

  • Experience with Java essential. Python, and R highly desired.
  • Experience with Apache Spark streaming and batch framework
  • Experience with Kafka, Storm, Zookeeper
  • Knowledge of Redis, Hadoop
  • Agile/Scrum methodology experience is required.
  • Service oriented architecture, and data standards like JSON, Avro, Protobuf
  • Strong communication skills and should be able to communicate effectively with business and other stake holders
  • Demonstrate ownership and initiative taking

Preferred Qualifications:

  • 5+ years of experience in application development with Java/J2EE, Spring, Spring Boot, Web Services, Scala, AWS
  • 2+ years of experience with Docker containerization
  • 3+ year of experience working with Big Data Technologies (Apache Storm, Apache Spark and Kafka)
  • 4+ years of experience in Unix, No SQL

Rvin James Murillo Andalan EA License No. 02C3423 Personnel Registration No. R1331697