The Client's Analytics team is looking for a strong technologist to help implement Spark/Scala jobs. The developer must not only know the Spark API well: They must also know about the pitfalls of distributed storage, how to structure a data processing pipeline that has to handle the 5V of Big Data-volume, velocity, variety, veracity, and value-and how to turn that into maintainable code.
- Help write analytics code, services and components in Java, Apache Spark, Kafka, Storm, Redis and related technologies such as Hadoop, Zookeeper
- Responsible for systems analysis - Design, Coding, Unit Testing and other SDLC activities
- Requirement gathering and understanding, Analyse and convert functional requirements into concrete technical tasks and able to provide reasonable effort estimates
- Spark query tuning and performance optimisation
- Work proactively, independently and with global teams to address project requirements, and articulate issues/challenges with enough lead time to address project delivery risks
- Providing expertise in technical analysis and solving technical issues during project delivery
- Code reviews, test case reviews and ensure code developed meets the requirements
- Experience with Java essential. Python, and R highly desired.
- Experience with Apache Spark streaming and batch framework
- Experience with Kafka, Storm, Zookeeper
- Knowledge of Redis, Hadoop
- Agile/Scrum methodology experience is required.
- Service oriented architecture, and data standards like JSON, Avro, Protobuf
- Strong communication skills and should be able to communicate effectively with business and other stake holders
- Demonstrate ownership and initiative taking
- 5+ years of experience in application development with Java/J2EE, Spring, Spring Boot, Web Services, Scala, AWS
- 2+ years of experience with Docker containerization
- 3+ year of experience working with Big Data Technologies (Apache Storm, Apache Spark and Kafka)
- 4+ years of experience in Unix, No SQL
Rvin James Murillo Andalan EA License No. 02C3423 Personnel Registration No. R1331697