Elvis: Building AI Workflows - Own the Orchestrator, own the Harness
- 16. Juni
- 2 Min. Lesezeit
Aktualisiert: vor 7 Tagen
Key Takeaways:
Elvis emphasizes the importance of owning the orchestrator and the harness for complex AI workflows
the orchestration, the harness, routing capabilities, dynamic artifacts/workflows, verifiers, ability to switch/route between agent backends, automations, the skills, and the MCP tools would be the absolute best defense for what happened with Fable this week.
The potential downside: high maintenance, might be too costly, and migth be unsustainable
The bigger your AI powered workflows become, the more complex, the harder they'll become to maintain in an ever changing landscape of new models and nerfed old models or decreased performance
Tran points out that without a verifying quality metric prompt optimizations e.g. for an llm classifier (such as intent classification) become vibe optimizing
She proposes using the DSPy package
DSPy turns that process into a repeatable workflow:
define the task,
evaluate it with examples,
optimize it with a metric, and
save the improved classifier.
As I see it, the question is how stable your workflows shall be - the more work you put into the orchestrator, verifiers, quality measures, the easier it will be to maintain quality for a more complex multi-agent setup
Especially for companies that want to provide high quality services, this is the way.
Elivs also points out Omnigent, an open source project to build harnesses
Real-time collaboration, multiple interfaces to the same agent, cloud execution, contextual security policies, cost policies, sandboxing, multi-harness authoring
But be aware: 26.1% of skills contain at least one vulnerability, spanning 14 distinct patterns across four categories: prompt injection, data exfiltration, privilege escalation, and supply chain risks (source: Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale)
Sources:
