Big Data Engineer
About The Position
Datorama’s platform consists of two main big building blocks – the data ingestion pipeline and the query/analysis pipelines.
The Big Data team deals with the main parts of the ingestion pipeline, as well as microservices and message bus infrastructure.
As a member of the team, you will be occupied with writing and architecting complex Spark based processes along with related microservices, and infrastructure components to enable reliability, monitoring, and overall scale of the system.
Things you’ll do:
- Use Spark to implement truly scalable ETL processes.
- Design next generation data lake, built for scale and based on Presto / Hive / Spark / S3 /Azure/ Vertica.
- Build autonomous microservices.
- 3+ years' experience developing and deploying robust, large-scale distributed systems in Java / Scala - Must.
- B.Sc. degree or higher in Computer Sciences.
- Experience with Big Data technologies (Hadoop echo system, Spark framework) - a big plus.
- Experience with SQL and NoSQL databases – a big plus.
- Proactive developer- strive to understand why things are done the way they are and don't take the present state for granted.
- Team player with strong communication skills.