Big Data Engineer
Search on its own is a highly complex data-set. Captify’s technologies have been built to extract maximum value from search for brands, partners and businesses all over the world and also to innovate & improve the consumer experience.
Captify’s world-class engineers, semantic specialists, and product teams are building the future of Search and as part of our Engineering Team you will play a key part in developing our offering.
Our Big Data team is enabling the optimal storage, organisation and timely retrieval of our data.
They are responsible for optimisation of ETL pipelines, maintaining over 60 Spark jobs. Building a data lake for data scientists and analysts.
We work in small cross-functional pods of developers who focus on a particular product areas and we encourage people to rotate across teams More detail around each of the team’s responsibilities can be found here.
Simon Edwards has joined us as our VP of Engineering and we are thrilled to have his experience on board, developing our teams and the future of engineering.
We’ve been growing rapidly and as we start our international roll-out, we’re looking for ambitious engineers to help us build the future of work.
With exciting projects, technologies and services in the pipeline now is a great time to be part of our journey
What you will be doing:
– You will help architect and build solutions to business-critical problems
– You will be participating in interesting projects such as:
– Carrying out efficient integration with our data providers via various API endpoints and data representation formats.
– Building and deploying an in-house distributed ETL pipeline for processing petabytes of data per day
– Enable an accurate, comprehensive and reliable data storage in our distributed data warehouses based on the needs of other teams
– Providing continuous improvements in the way data is being processed and stored based on the feedback and needs of the business or other teams
– Setting up monitoring for key performance metrics and overall systems’ behaviour to promptly react in case any anomaly detected
– You will be responsible for optimisation of ETL pipelines, maintaining over 60 Spark jobs. Building a data lake for data scientists and analysts.
– Experimenting with new tools and technologies to produce cutting-edge solutions to business problems
– Be a part of a self-organising, results-oriented agile team using Kanban to complete new product launches
What you need to be successful:
We’re looking for a mid to senior-level candidate who has opinions but wants to do the right thing; we want you to own the product, and care about the direction it goes in
– Have ~5 years experience in software development and have exceptional problem-solving skills
– Have a strong desire to improve your skills with modern development and production environment – Unix/Linux, bash, python, virtualisation (docker, Kubernetes),
– Have deployment orchestration (chef, puppet, ansible), configuration servers ZooKeeper, monitoring and distributed logging (Prometheus, Grafana, GrayLog) experience
– Have strong knowledge of Java or any JVM based language
– Have familiarity with distributed data processing frameworks Hadoop, Spark, Kafka.
– Have proven experience of building production-ready ETL processing pipelines
– Have a proven track record of building integration with third parties via various APIs (http, file, message base)
– Have familiarity with Big Data persistent distributed storages Impala, Elasticsearch, Aerospike, HBase
– Have proven working record with relational databases (e.g. PostgreSQL (stored functions, triggers))
– Have good written and verbal English communication skills
You earn bonus points for the following:
– Scala knowledge
– Experience in AWS S3 environment (S3 storage)
– Some experience in the AdTech sector is advantageous
– Previous experience in Bank or any Accounting field is a strong advantage
– Experience with working in an international team
– A B.Sc. in Computer Science, or related qualification
What is in it for you:
Flexible working – We give you the option to Work from Home or better still you can work from any Captify office for 2 weeks in total per year.
Sharing culture – If you have learned something new, we welcome you sharing to the to share it with the team through a short presentation.
Diversity – You will be working in global diverse teams with intelligent and like minded individuals
Days off – You will get 20 days holiday per year plus 2 Emergency days off and your birthday. As well as all 11 of the Ukraine public holidays off
Limited Company – We help manage your Limited Company and provide you with supporting where needed
Captify is an equal opportunities employer. We are determined to create a diverse group at all levels of our Company, and we welcome all members of the community to apply for openings with us.