Devin just came to take your software job… will code for $8/hr

Summary

Devon, an AI tool that automates junior engineering tasks, has arrived and is shaking up the software industry. For just $500/month, it promises to outperform 74.2% of human programmers by building, testing, and shipping code autonomously. While the technology is impressive, it’s not without flaws—Devon struggles with less popular tools and occasionally generates buggy code. Despite its quirks, the tool’s integration with Slack and GitHub offers a compelling workflow for developers, especially for enterprise teams. While the future of AI in programming looks promising, Devon's adoption comes with growing pains, as it challenges traditional workflows and expectations.

Takeaways

• 😀 Devon is a fully automated junior engineer AI that can build, test, and ship code, outperforming 74.2% of human software engineers.
• 😀 The price for Devon is $500 per month, or about $8 per hour based on its usage model called Agent Compute Units.
• 😀 Devon's functionality is accessed via Slack, targeting non-programmers and enterprise workers who primarily use Slack for communication.
• 😀 Despite being an AI tool, Devon still suffers from quirks such as unnecessary dependencies and occasional hallucinations when explaining code changes.
• 😀 While Devon works well with mainstream technologies like ReactJS, it struggles with more niche tools, like Spel.
• 😀 Devon can automate the entire software development lifecycle, from writing code to pushing it to GitHub, but it's not yet perfect and can produce bugs or make unintended code changes.
• 😀 Cognition Labs, the company behind Devon, raised $175 million at a $2 billion valuation, but it has no revenue yet, and Devon’s performance is largely unproven.
• 😀 There are no scientific benchmarks or published research behind Devon’s capabilities, leaving its performance claims mostly unsupported by rigorous data.
• 😀 Devon faced a security issue early on when a member exposed a VS Code live share URL, allowing unauthorized access, but it was quickly fixed with the help of its own tools.
• 😀 The major downside of Devon's design is its Slack-based workflow, which is seen as outdated and less appealing to developers used to more modern tools.
• 😀 The AI tool landscape is growing, with competing products from OpenAI, Google, and others, but tools like Devon are positioned to disrupt the software engineering field.
• 😀 Despite its flaws, Devon represents a significant leap forward in AI for software development, demonstrating what might be possible with future advancements in automation and machine learning.

Q & A

What is Devon, and how does it relate to software engineers?

Devon is an AI-powered tool designed to automate the tasks of junior software engineers, including writing, testing, and shipping code. It can perform these tasks more efficiently than 74.2% of humans, leading to fears among software engineers that their jobs could be at risk. Devon offers a cheaper alternative to hiring human developers, priced at $500 per month.

How does Devon's pricing model work?

Devon's pricing is based on 'Agent Compute Units,' which translates to about $8 per hour. This pricing structure is relatively affordable for software engineers, though it is a bit vague about the specifics of what exactly Devon is capable of, making it unclear whether it's truly worth the cost.

Why is there skepticism about Devon's effectiveness?

Devon is new, and there's skepticism because it lacks benchmarks, scientific papers, or clear performance metrics. While its creators claim it works better than other AI tools, such as OpenAI's models, there is no independent verification of these claims, raising doubts about its true capabilities.

What are some of the limitations of Devon as noted in the video?

Devon struggles with more complex or lesser-known programming tools, like Spel, and can make errors in code, such as introducing unnecessary packages or changing unrelated parts of the code when fixing bugs. It also tends to hallucinate explanations when asked for clarifications on its decisions.

How does Devon interact with users, and what is the user experience like?

Devon is designed to work through Slack, which may be an unconventional choice for developers who are used to more direct interfaces. Users interact with Devon by tagging it in Slack to assign coding tasks, and it creates a workspace with a shell, browser, and editor to complete the task.

What is the significance of Devon's ability to create pull requests on GitHub?

Devon's ability to create pull requests on GitHub is a key feature, as it enables seamless integration with development workflows. After completing tasks, Devon can submit its code directly to GitHub, allowing for collaboration and integration into existing development pipelines.

How does Devon compare to other AI tools like OpenAI's GPT models?

While Devon is capable of generating code, it is designed specifically for non-programmers and enterprises that need to automate software engineering tasks. Unlike OpenAI's models, which are broader in scope, Devon focuses solely on writing and shipping code, with mixed results depending on the complexity of the task.

What is the 'trust me bro' benchmark mentioned in the video?

The 'trust me bro' benchmark refers to the informal and unscientific performance claims made by Devon's creators. These benchmarks lack independent validation and rely on internal testing, which raises doubts about the true capabilities of Devon and whether it can live up to the hype.

Why is the video creator concerned about Devon's potential impact on software engineers?

The video creator expresses concern that tools like Devon could make junior software engineers obsolete by automating their tasks at a fraction of the cost. This could lead to widespread job losses in the software engineering industry, particularly for developers working at the entry level.

What is PGI, and how does it relate to AI development?

PGI, created by TimeScale, is a tool designed for working with time-series data in databases, built on top of PostgreSQL. The video creator mentions PGI in contrast to Devon, highlighting its usefulness in building AI systems like recommendation engines. PGI offers performance improvements and simplifies the development process for AI systems.