small benchmark im trying to run- does letting AI agents orchestrate themselves perform better'
or does it just add cost and more failures'

I built a quick benchmark harness along with 4 agent architectures to evaluate on:
- single: one agent with all tools
- explicit: a hand-authored workflow / DAG (deterministic)
- self-orch: a coordinator that decides at runtime which specialists to delegate to (A2A)
- hybrid: self-orch + a guardrail (a cap on delegations)

the setup kind of is:
-graded on ''-bench (Sierra's public tool-agent benchmark) with pass^k reliability [btw first 720 runs on my labelled task suite]
- scoring follows BFCL for tool-selection
- and the Inspect AI dataset'solver'scorer pattern
- local models on Ollama for NOW

I will be running the same four strategies on real AWS serverless architectures since i was originally experimenting as i will be speaking on this topic at AWS Community Day Bangalore

The TLDR till now from the base experiment (720 runs, 3 open models):
- The scripted workflow was more reliable than the autonomous one on every model, +58 points on the weakest (paired bootstrap, p'0)
- On an open-ended task, the unguarded self-orchestrator kept delegating until it hit its cap (6 hops); a simple guardrail held it to 2
- It costs more per query for less reliability, and the trade gets worse the weaker the model.

The read: self-orchestration is a tradeoff, for now

well it largely depends on the models and harness too. planning to take this further, primarily on AWS by using better inference models and I will setup AWS architectures for each workflow

and then locally for long runs with better models after i delete some games to free up space lol

you can check the bench here: lnkd.in/gZE83StR

I am trying to learn how do we even benchmark workflows in the first place. Honestly, just experimenting but its been fun. If yk a standard practice to it, please comment through or dm