The Startup Selling ‘AI Employees’ Just Raised $180M. We Tried Their Product.

Cognition AI, the company behind the Devin software engineering agent, closed a $180 million Series B last month at a $2 billion valuation. The pitch is straightforward: AI employees that work autonomously, report into your existing project management tooling, and cost a fraction of a human hire.

We spent three weeks putting Devin and two of its closest competitors — Factory and SWE-agent — through a structured evaluation. The results are more nuanced than either the hype or the backlash suggests.

On well-scoped, clearly specified tasks — “add input validation to this API endpoint,” “write unit tests for this module,” “migrate this function from Python 2 to 3” — Devin completes them correctly about 68% of the time without human intervention. That number drops to 41% for tasks requiring architectural judgment, and to 22% for tasks involving ambiguous requirements.

The honest business case is not “replace your engineers.” It is “handle the well-defined, low-ambiguity tail of your engineering backlog without pulling a senior engineer off higher-value work.” That’s a real value proposition. It is also a much smaller addressable market than the $2 billion valuation implies.