Data Platform Engineer
Hybrid
Full Time
#Technology
#AWS
#GCP
#Kafka
#Spark
#Kubernetes
What you will be doing:
Are you a highly skilled Data Platform Engineer with a passion for cutting-edge technology and a desire to make a real impact? Join our innovative team at ComplyAdvantage, where you'll design, manage, monitor, and optimise a diverse array of data systems running on AWS and GCP. Your expertise will ensure the seamless operation of SQL, NoSQL, Kafka, and Spark-based data layers, supporting our global client base.
Key Responsibilities:
- Innovative Design and Maintenance: Develop and sustain data layer orchestration systems, managing self-managed and PaaS data layer technologies, and ensure data consistency while designing databases.
- Performance and Availability: Oversee data layer components on AWS and GCP, guaranteeing top performance and reliability. Conduct regular performance tuning and troubleshoot issues.
- Backup and Cost Management: Configure and deploy robust backup systems for databases, Kafka, and Spark, ensuring data integrity and enabling disaster recovery. Implement cost management strategies to optimise resource utilisation.
- Documentation and Collaboration: Maintain thorough documentation for all Data Infrastructure engineering processes. Work closely with various teams to integrate data sources and support application requirements, adhering to best practices.
- Capacity and Project Planning: Monitor and manage data infrastructure capacity, planning for future growth and scalability. Collaborate with stakeholders to define and prioritise engineering projects, setting realistic timelines and expectations.
- Technical Support: Provide guidance and support to engineering team members on database and data layer architecture and technologies. Collaborate with Software Engineers to develop resilient data sharding strategies.
Our Tech Stack:
- Cloud-Based Infrastructure: Fully cloud-based with a Kubernetes-focused tech stack. Compute workloads run in Kubernetes clusters across multiple regions.
- Infrastructure Management: Heavy use of Terraform and Helm, adhering to GitOps paradigms for managing cloud infrastructure and Kubernetes applications.
- Core Technologies: Extensive use of Kafka, distributed PostgreSQL and Cassandra QL, Elasticsearch, and Databricks/Spark. Development of inter-cloud failover options to support multi-cloud plans.
- Wide Array of Applications: Teams build and release containerised applications for low latency APIs, machine learning models, and data processing pipelines.
About You:
- Experience: Significant expertise with AWS and/or GCP core services, data systems (Apache Kafka, Apache Spark, Elasticsearch, PostgreSQL, Cassandra), and Kubernetes. Experience codifying infrastructure in Terraform and/or Helm charts.
- Cost and Performance: Awareness of cost and performance considerations in designing data systems architectures. Proficiency with logging, monitoring, and tracing tools.
- CI/CD and Security: Familiarity with CI tooling, test pipelines, and asset generation (e.g., Docker images, Helm charts). Understanding of security considerations in data systems.
- Data Modelling: Experience with data modelling, sharding, and indexing strategies for large-scale databases.
Nice to Have:
- Experience running distributed databases within Kubernetes.
- Familiarity with distributed SQL and NoSQL databases such as Yugabyte, Cockroach, Spanner, HBase, or CouchDB.
Education:
- BSc/BA degree in computer science, engineering, or related discipline OR equivalent experience in required skills
What’s in it for you?
- Equity as we want you to have a part of what we are building
- Unlimited Time Off Policy- A work-life balance and focus on our well-being are critical to keeping us performing at our best
- We embrace a hybrid approach that requires employees to be in the office for two days a week. We strongly believe that this approach fosters collaboration and enables the building of meaningful relationships
- You will also get a new starter budget to kit out your home office
- Opportunity to work on innovative projects with smart-minded people keen to share their knowledge and continuously improve
- Annual learning budget (prorated based on start date) to drive your performance and career development
About us:
ComplyAdvantage is the financial industry’s leading source of AI-driven financial crime risk data and detection technology. Our mission is to neutralise the risk of money laundering, terrorist financing, corruption, and other financial crime.
More than 1000 companies rely on us to understand the risk of who they’re doing business with through the world’s only global, real-time database of people and companies. Our solutions identify thousands of risk events daily from millions of structured and unstructured data points.
We have five global hubs in New York, London, Singapore, Lisbon and Cluj-Napoca and are backed by Goldman Sachs, Ontario Teachers, Index Ventures, and Balderton Capital.
Since 2014, we have raised over $100 million in funding, and in 2022 alone grew by over 40% to over 500 people globally. Over the next 12 months, as our revenue increases, we plan to increase to 600.
At ComplyAdvantage diversity fuels our rocket ship and our commitment to inclusion across race, gender, age, religion, identity and experience drives us forward every day. We encourage everyone to apply and aspire to consider every application fairly.
We will handle your information in accordance with our Privacy Policy. For further information, please click here.
