Skip to content
trendoviz

Together AI reports 28% higher B200 throughput

Together AI says Together Compiler uses AI agents to rewrite and fuse GPU kernels, reaching 2,118 tokens/s versus 1,649 tokens/s for a hand-written megakernel on an NVIDIA B200 running Llama 3.2 1B. These figures come from the company’s own blog and have not been independently verified.

47 out of 100Emerging7D: Insufficient history ProvisionalPublished

Overview

Together AI has published the direction of Together Compiler, a framework for continual AI inference optimization built around AI agents that rewrite and fuse GPU kernels in a model’s forward pass [1]. The company’s results point to three related ideas rather than a single replacement for megakernels:

  1. Standard compiler features. Applying CUDA Graphs and PDL to default Torch Inductor kernels recovered 98% of megakernel performance in one reported experiment, without custom kernel code [1].
  2. A multi-kernel design. Together AI says it decomposed the earlier approach into “multi-kernels,” retaining separate kernels rather than placing an entire forward pass inside one fused kernel [1].
  3. Agent loops that revise kernels. Agents iteratively modify the kernels, and the company reports that combining them with multi-kernels beat its earlier megakernels [1].

Together AI positions the framework as a way to reduce recurring kernel-engineering work as models, traffic patterns and hardware change. The retrieved text does not document the agent workflow, validation process or production deployment in enough detail to assess that claim, and it does not explain what PDL stands for [1].

Why it matters

The reported B200 result suggests that agent-generated kernels may offer more than a modest tuning gain over a hand-written megakernel: Together AI puts the difference at 28%, or 2,118 versus 1,649 tokens/s for Llama 3.2 1B [1]. For serving teams, the more consequential claim may be operational. Together AI says its earlier megakernels took months of work and reached production with only one model, one batch size and one customer [1]. A repeatable optimization process could, in principle, fit changing model and traffic configurations better than bespoke kernel engineering. That remains a vendor-stated rationale rather than a demonstrated outcome, because the evidence does not show how the production system works [1].

The 98% recovery figure adds context to the agent results. If standard compiler features account for most of the earlier performance, then the value of agents would depend on how much headroom remains after graphs, PDL and multi-kernel grouping are applied [1]. The retrieved material does not isolate that contribution.

Opportunities

Developers could evaluate agent-generated kernels as an alternative to maintaining a fully hand-written megakernel when architectures or hardware targets change frequently [1]. The multi-kernel design also suggests a different optimization target: adjust how separate kernels are grouped and scheduled instead of collapsing a forward pass into one kernel [1].

Tool builders could focus on the gaps the published results leave open, including numerical-equivalence checks against a baseline, repeatable benchmarking across batch sizes and sequence lengths, and rollback mechanisms for generated kernels [1].

Risks

Agent-generated kernel changes may not preserve numerical equivalence, and the available material gives no account of how outputs are verified against a baseline [1].

The approach also depends on CUDA-specific mechanisms such as CUDA Graphs and PDL, which may limit portability across accelerator stacks [1]. Vendor-reported speedups create a comparison risk because the retrieved text does not specify benchmark conditions such as batch size, sequence length, numerical precision or exact model configurations [1].

Evaluation checklist for a serving team

  • Correctness validation: How are generated kernels checked for numerical equivalence with the current stack?
  • Batch-size and precision coverage: Which combinations are tested, and how do results vary?
  • End-to-end latency: Does throughput translate into lower response latency for actual traffic?
  • Compilation and agent costs: What time, compute and cost do agent loops add?
  • Portability: Does the approach work beyond CUDA and the tested hardware?
  • Rollback behavior: Can a problematic generated kernel be reverted without a new release?

Context

Torch Compile provides the baseline against which Together AI frames its work. TorchDynamo traces Python execution into an FX graph using fake tensors that carry metadata without allocating or computing tensor data. Inductor lowers that graph into categories including pointwise operations, reductions, scans and templates, while operations such as GEMMs may use external libraries such as cuBLAS. Its fusion machinery generally treats those external kernels as opaque [1].

Inductor uses a greedy pairwise process: can_fuse() checks whether a fusion is legal, and score_fusion() ranks candidates using heuristics such as estimated memory-traffic savings. These decisions are local rather than a global search over kernel partitions and execution strategies [1].

Together AI reports that Inductor delivered 669 tokens/s on its target Llama 1B workload and says its heuristics struggle on other architectures and newer hardware [1]. The company’s answer combines existing compiler mechanisms, a multi-kernel structure and agent-driven revision. The published material also refers to “40+ Models Chosen for Production,” but the retrieved text does not identify those models or describe the production system [1].

Known, claimed, inferred

Together AI’s own account explains why it moved away from a single megakernel: the fused approach was difficult to deploy broadly, while Torch Inductor and CUDA mechanisms already capture much of the performance [1]. Its proposed extension adds multi-kernels and agent loops, and the company reports gains of up to 1.7x over its earlier megakernels, with more than 1,100 tokens/s for Llama 1B on H100 [1].

Those gains should not be attributed to agents alone. The retrieved text presents three variables that may contribute: standard compiler features, multi-kernel grouping and agent revision [1]. It does not provide an ablation or specify how much each contributes, nor does it explain the agent workflow or validation process in enough detail to assess the claimed reduction in recurring engineering work [1].

Reported results

MethodModelHardwareReported throughputComparison baselineStatus
Default Torch InductorLlama 1B (target workload)Not specified669 tokens/sNone statedVendor-reported result
Default Torch Inductor with CUDA Graphs and PDLNot specifiedNot specified98% of megakernel performance recoveredEarlier hand-written megakernelVendor-reported result
Multi-kernels with agent loopsLlama 1BH100More than 1,100 tokens/sEarlier megakernels, up to 1.7xVendor-reported result
Agent-generated kernelsLlama 3.2 1BNVIDIA B2002,118 tokens/sHand-written megakernel, 1,649 tokens/s; 28% higherVendor-reported result

These are separate reported experiments, not one continuous benchmark. The 98% recovery result concerns CUDA Graphs and PDL applied to default Inductor kernels. The H100 result concerns multi-kernels with agent loops on Llama 1B. The B200 comparison concerns agent-generated kernels on Llama 3.2 1B against a hand-written megakernel [1]. The rows share a model family and a common line of investigation, but the evidence does not establish that they used matching workloads, conditions or baselines [1].

Limitations & uncertainties

All performance figures come from Together AI’s own blog. No independent benchmark, third-party test report or user account is available in the retrieved material [1]. The source article is truncated before its later sections, leaving out the described details of the agents, multi-kernels, production system and expansion of PDL [1].

There are no measurements for compilation overhead, agent execution cost, end-to-end latency or the cost of producing the reported throughput. No evidence describes correctness checks for generated kernels or identifies the models included in the referenced production count [1].

Evidence & sources

The key statements of this analysis, labeled by type, with the sources they rely on. Trendoviz does not run product tests.

  • Primary-source or vendor statementTogether AI says Together Compiler uses AI agents to rewrite and fuse GPU kernels in a model’s forward pass.[1]
  • Primary-source or vendor statementTogether AI reports 2,118 tokens/s for Llama 3.2 1B on NVIDIA B200, compared with 1,649 tokens/s for a hand-written megakernel, a 28% difference.[1]
  • Primary-source or vendor statementTogether AI reports that CUDA Graphs and PDL applied to default Torch Inductor kernels recovered 98% of megakernel performance without custom kernel code.[1]
  • Primary-source or vendor statementTogether AI reports that multi-kernels with agent loops beat its earlier megakernels by up to 1.7x, achieving more than 1,100 tokens/s for Llama 1B on H100.[1]
  • Primary-source or vendor statementTogether AI reports that Torch Inductor delivered 669 tokens/s on its target Llama 1B workload.[1]
  • Primary-source or vendor statementTorchDynamo traces Python execution into an FX graph using fake tensors, and Inductor lowers operations into categories including pointwise operations, reductions, scans, templates and opaque external kernels.[1]
  • Primary-source or vendor statementInductor’s fusion process uses greedy pairwise decisions based on local heuristics rather than a global search over kernel partitions and execution strategies.[1]
  • Primary-source or vendor statementTogether AI says its earlier megakernels took months of effort and reached production with one model, one batch size and one customer.[1]

Sources

  1. 1

Updates & corrections

  1. First published · approved by an editor

Found an error? Report it.