- 6 min read
- 8 key claims
- 1 source cited
- For Developers, Businesses
Overview
Together AI has published the direction of Together Compiler, a framework for continual AI inference optimization built around AI agents that rewrite and fuse GPU kernels in a model’s forward pass [1]. The company’s results point to three related ideas rather than a single replacement for megakernels:
- Standard compiler features. Applying CUDA Graphs and PDL to default Torch Inductor kernels recovered 98% of megakernel performance in one reported experiment, without custom kernel code [1].
- A multi-kernel design. Together AI says it decomposed the earlier approach into “multi-kernels,” retaining separate kernels rather than placing an entire forward pass inside one fused kernel [1].
- Agent loops that revise kernels. Agents iteratively modify the kernels, and the company reports that combining them with multi-kernels beat its earlier megakernels [1].
Together AI positions the framework as a way to reduce recurring kernel-engineering work as models, traffic patterns and hardware change. The retrieved text does not document the agent workflow, validation process or production deployment in enough detail to assess that claim, and it does not explain what PDL stands for [1].
Why it matters
The reported B200 result suggests that agent-generated kernels may offer more than a modest tuning gain over a hand-written megakernel: Together AI puts the difference at 28%, or 2,118 versus 1,649 tokens/s for Llama 3.2 1B [1]. For serving teams, the more consequential claim may be operational. Together AI says its earlier megakernels took months of work and reached production with only one model, one batch size and one customer [1]. A repeatable optimization process could, in principle, fit changing model and traffic configurations better than bespoke kernel engineering. That remains a vendor-stated rationale rather than a demonstrated outcome, because the evidence does not show how the production system works [1].
The 98% recovery figure adds context to the agent results. If standard compiler features account for most of the earlier performance, then the value of agents would depend on how much headroom remains after graphs, PDL and multi-kernel grouping are applied [1]. The retrieved material does not isolate that contribution.
Opportunities
Developers could evaluate agent-generated kernels as an alternative to maintaining a fully hand-written megakernel when architectures or hardware targets change frequently [1]. The multi-kernel design also suggests a different optimization target: adjust how separate kernels are grouped and scheduled instead of collapsing a forward pass into one kernel [1].
Tool builders could focus on the gaps the published results leave open, including numerical-equivalence checks against a baseline, repeatable benchmarking across batch sizes and sequence lengths, and rollback mechanisms for generated kernels [1].
Risks
Agent-generated kernel changes may not preserve numerical equivalence, and the available material gives no account of how outputs are verified against a baseline [1].
The approach also depends on CUDA-specific mechanisms such as CUDA Graphs and PDL, which may limit portability across accelerator stacks [1]. Vendor-reported speedups create a comparison risk because the retrieved text does not specify benchmark conditions such as batch size, sequence length, numerical precision or exact model configurations [1].
Evaluation checklist for a serving team
- Correctness validation: How are generated kernels checked for numerical equivalence with the current stack?
- Batch-size and precision coverage: Which combinations are tested, and how do results vary?
- End-to-end latency: Does throughput translate into lower response latency for actual traffic?
- Compilation and agent costs: What time, compute and cost do agent loops add?
- Portability: Does the approach work beyond CUDA and the tested hardware?
- Rollback behavior: Can a problematic generated kernel be reverted without a new release?
Context
Torch Compile provides the baseline against which Together AI frames its work. TorchDynamo traces Python execution into an FX graph using fake tensors that carry metadata without allocating or computing tensor data. Inductor lowers that graph into categories including pointwise operations, reductions, scans and templates, while operations such as GEMMs may use external libraries such as cuBLAS. Its fusion machinery generally treats those external kernels as opaque [1].
Inductor uses a greedy pairwise process: can_fuse() checks whether a fusion is legal, and score_fusion() ranks candidates using heuristics such as estimated memory-traffic savings. These decisions are local rather than a global search over kernel partitions and execution strategies [1].
Together AI reports that Inductor delivered 669 tokens/s on its target Llama 1B workload and says its heuristics struggle on other architectures and newer hardware [1]. The company’s answer combines existing compiler mechanisms, a multi-kernel structure and agent-driven revision. The published material also refers to “40+ Models Chosen for Production,” but the retrieved text does not identify those models or describe the production system [1].
Known, claimed, inferred
Together AI’s own account explains why it moved away from a single megakernel: the fused approach was difficult to deploy broadly, while Torch Inductor and CUDA mechanisms already capture much of the performance [1]. Its proposed extension adds multi-kernels and agent loops, and the company reports gains of up to 1.7x over its earlier megakernels, with more than 1,100 tokens/s for Llama 1B on H100 [1].
Those gains should not be attributed to agents alone. The retrieved text presents three variables that may contribute: standard compiler features, multi-kernel grouping and agent revision [1]. It does not provide an ablation or specify how much each contributes, nor does it explain the agent workflow or validation process in enough detail to assess the claimed reduction in recurring engineering work [1].
Reported results
| Method | Model | Hardware | Reported throughput | Comparison baseline | Status |
|---|---|---|---|---|---|
| Default Torch Inductor | Llama 1B (target workload) | Not specified | 669 tokens/s | None stated | Vendor-reported result |
| Default Torch Inductor with CUDA Graphs and PDL | Not specified | Not specified | 98% of megakernel performance recovered | Earlier hand-written megakernel | Vendor-reported result |
| Multi-kernels with agent loops | Llama 1B | H100 | More than 1,100 tokens/s | Earlier megakernels, up to 1.7x | Vendor-reported result |
| Agent-generated kernels | Llama 3.2 1B | NVIDIA B200 | 2,118 tokens/s | Hand-written megakernel, 1,649 tokens/s; 28% higher | Vendor-reported result |
These are separate reported experiments, not one continuous benchmark. The 98% recovery result concerns CUDA Graphs and PDL applied to default Inductor kernels. The H100 result concerns multi-kernels with agent loops on Llama 1B. The B200 comparison concerns agent-generated kernels on Llama 3.2 1B against a hand-written megakernel [1]. The rows share a model family and a common line of investigation, but the evidence does not establish that they used matching workloads, conditions or baselines [1].
Limitations & uncertainties
All performance figures come from Together AI’s own blog. No independent benchmark, third-party test report or user account is available in the retrieved material [1]. The source article is truncated before its later sections, leaving out the described details of the agents, multi-kernels, production system and expansion of PDL [1].
There are no measurements for compilation overhead, agent execution cost, end-to-end latency or the cost of producing the reported throughput. No evidence describes correctness checks for generated kernels or identifies the models included in the referenced production count [1].
Evidence & sources
The key statements of this analysis, labeled by type, with the sources they rely on. Trendoviz does not run product tests.
- Primary-source or vendor statementTogether AI says Together Compiler uses AI agents to rewrite and fuse GPU kernels in a model’s forward pass.[1]
- Primary-source or vendor statementTogether AI reports 2,118 tokens/s for Llama 3.2 1B on NVIDIA B200, compared with 1,649 tokens/s for a hand-written megakernel, a 28% difference.[1]
- Primary-source or vendor statementTogether AI reports that CUDA Graphs and PDL applied to default Torch Inductor kernels recovered 98% of megakernel performance without custom kernel code.[1]
- Primary-source or vendor statementTogether AI reports that multi-kernels with agent loops beat its earlier megakernels by up to 1.7x, achieving more than 1,100 tokens/s for Llama 1B on H100.[1]
- Primary-source or vendor statementTogether AI reports that Torch Inductor delivered 669 tokens/s on its target Llama 1B workload.[1]
- Primary-source or vendor statementTorchDynamo traces Python execution into an FX graph using fake tensors, and Inductor lowers operations into categories including pointwise operations, reductions, scans, templates and opaque external kernels.[1]
- Primary-source or vendor statementInductor’s fusion process uses greedy pairwise decisions based on local heuristics rather than a global search over kernel partitions and execution strategies.[1]
- Primary-source or vendor statementTogether AI says its earlier megakernels took months of effort and reached production with one model, one batch size and one customer.[1]
Sources
- 1Last year we built megakernels. This year we deleted them: The road to Together Compiler●together.airetrieved Nov 18, 2026
Updates & corrections
- First published · approved by an editor
Found an error? Report it.