Senior Data Engineer - Consensus｜Meet.jobs

Company Overview:

Scientific research contains the most valuable knowledge humans have ever created. Yet, there’s no easy way to search and consume it.

This is the problem the Consensus team is obsessed with solving. Our mission is to democratize access to the world’s best knowledge.

Today, Consensus is an AI-powered academic search engine. We use large language models to find and analyze answers in scientific research papers. We serve scientific researchers, students (high schoolers to PhDs), doctors healthcare workers, industry R&D teams and the science-curious.

Try a search in Consensus for yourself: https://consensus.app/

Company Highlights:

• Since launching in the Winter of 2022, we’ve grown to over 400,000 monthly active users and over $1M in annual recurring revenue - up over 600% in 2024.
• $15M in total funding from some of the top AI investors in the world, including USV, Nat Friedman, Daniel Gross, and Draper Associates
• Our small but mighty team of 10 (and growing) includes alumni of top-tier tech companies, including Amazon, Google, Quora, PayPal, Waymo, and more.
• Consensus has been featured in The Wall Street Journal, The Atlantic, The New York Times, The Boston Globe, Nature, a16z, and is recognized as one of the most exciting new AI search engines in the world.

The role: Building the search engine for the future of research is a massive data challenge, so we are looking for an experienced data engineer to join our team! You will strengthen and streamline the sourcing, ingestion, and orchestration of our data. Some core tasks:

• Increase the frequency and automation of our dataset updates
• Manage the canonicalization of data derived from multiple sources
• Orchestrate and optimize pipelines processing raw and ML generated feature datasets
• Source new datasets, possibly via crawling
• Opinionated development of event based and batch processing pipelines
• Architect and manage data and feature stores for easy access and introspection of our core data

What we value:

• 5+ years working on challenging data engineering problems
• Prefer simple solutions and have a good intuition for what will scale
• Familiar with cloud infrastructure and modern data pipeline frameworks (Spark, Beam)
• Enjoy writing in typed Python, following best practices, and proactively updating and maintaining the codebase
• Collaborative team player who makes decisions based on what’s best for the mission
• You care about creating a more scientifically-informed world and democratizing access to knowledge through technology

Nice to haves:

• Experience in ML infra, search infra, distributed systems, and dev-ops
• Experience/interest in search engine applications or web crawling

Compensation:

• $180-$250k cash
• Competitive Series A equity

Final offer amounts are determined by multiple factors, including, experience and expertise, and may vary from the amounts listed above.