Discussion about this post

User's avatar
facile's avatar

> The factory of the future isn’t a coding agent, an IDE plugin, or a model API. It’s a full-stack service that accepts a spec from a brand and delivers running software continuously. That means model orchestration, code creation, hosting, implementation, testing, deployment, monitoring, maintenance, evolution. End to end.

In a sense, this would essentially be a cloud provider with a dedicated LLM interface that can handle both resource provisioning and change management.

The thing is: software development is already pretty close to this. If Amazon owned GitHub, then one could imagine an end-to-end software factory where devs just push spec changes and let it rip.

The problem comes down to the granularity of the specification. For Nike, their designers already had a complete picture of what the shoe should look like, how it should feel on your feet, what materials should be used, etc. They're full stack cobblers.

But, in software, the product spec is not sufficient detailed enough to accurately describe the technical specifications of the design. LLMs can make decisions to fill in the gaps, but they don't yet seem to be able to fill the gap between an ambiguous product spec and a firm technical design that supports a resilient architecture in the long term. And I don't think this is an IQ problem: making the models smarter does not seem to necessarily help them build a stronger intuition for this. There's a fundamental ambiguity to plain-language product specifications that takes a honed intuition to successfully interpret into code.

Also, even big IQ models still seem to struggle with maintaining a coherent architecture due to the size of their context windows. Even the biggest models can't hold an entire distributed system in their context windows, and often start to write more bugs as the system grows in complexity.

Perhaps future models will solve all of these problems -- but perhaps not.

Aspiring FIRE's avatar

This is well written but imo over optimistic. The base issue of hallucination and determining the quality of any model output hasn't been solved yet.

18 more comments...

No posts

Ready for more?