Carbon-Aware 3D Parallelism for LLM Training

Training frontier LLMs is expensive. Like, thousands of GPUs running for weeks expensive. Those GPUs are pulling power from the grid, so the training run has a carbon cost too. The usual systems instinct is to push parallelism as far as possible and minimize wall-clock time. That makes sense if time is the only objective, but it leaves the carbon side mostly out of the optimization.

This project started from a pretty simple observation: throughput scales sublinearly with data parallelism, but carbon emissions scale nearly linearly with the number of active devices. That asymmetry is the part I kept coming back to.

Quick Primer on 3D Parallelism

When you train a huge model across a GPU cluster, you're typically combining three types of parallelism:

1. Data Parallelism (DP): replicate the model across devices, each processes a different chunk of data, then you sync gradients. This is the easiest to scale but communication overhead grows as you add more replicas.
2. Tensor Parallelism (TP): split individual layers across devices. Great for large layers but requires fast interconnects since devices are constantly communicating within a single forward pass.
3. Pipeline Parallelism (PP): split the model into stages and pipeline microbatches through them. Introduces "bubble" overhead where some stages are idle waiting for others.

Systems like Megatron-LM combine all three to train models at massive scale. The standard goal is to find the (DP, TP, PP) configuration that minimizes training time. Tools like Galvatron and InternEvo automate this search.

The Asymmetry

Here's where it gets interesting. When you double DP, you roughly double your power consumption because you're activating twice as many GPUs. But you don't double your throughput, because gradient synchronization overhead eats into your gains. The more DP replicas you add, the worse this ratio gets.

So you end up in a situation where the last chunk of data parallelism you add is giving you a small bump in tokens/sec but costing you a proportionally large bump in carbon. That's wasteful, especially during times when the electrical grid is running dirty (high carbon intensity).

The Idea

The carbon intensity of the grid isn't constant. It fluctuates throughout the day depending on the energy mix: solar, wind, natural gas, coal, etc. So the same computation emits very different amounts of carbon depending on when you run it.

Our approach is a carbon-aware controller that dynamically adjusts the (DP, TP, PP) configuration over time based on carbon intensity forecasts. The logic is straightforward:

When carbon intensity is high, we scale back DP to reduce the number of active GPUs and retune TP and PP to keep utilization reasonable on the remaining devices. When carbon intensity drops, we scale DP back up to recover throughput and make progress faster.

The optimization objective minimizes total emissions $\int_0^T P(\pi_t) \cdot CI(t) \, dt$ subject to completing the required training workload $W$. At each decision epoch the controller observes a carbon intensity forecast, estimates throughput and power for candidate configurations, and solves a constrained optimization to pick the best one.

What We Expect

The bet is that we can reduce emissions by something like 20-40% while keeping the slowdown under 10%. That tradeoff feels worth taking seriously. In many training runs, a small increase in wall-clock time may be a reasonable price to pay if it avoids running a large cluster at the dirtiest parts of the day.

We're benchmarking against three baselines: a fixed time-optimized 3D configuration, a simpler approach that only adjusts DP without retuning TP/PP, and carbon-aware workload shifting (GreenFlow style) that moves entire jobs to low-carbon periods without changing parallelism. The key metrics are total CO2 emissions, wall-clock training time, and carbon efficiency measured in tokens per kgCO2.

Why This Matters

The conversation around AI sustainability often focuses on measuring emissions after the fact or buying offsets. That work is useful, but it does not say much about how to change the training run itself. This project is about intervening at the systems level, where the parallelism decisions are already being made.

The useful part is that the idea does not require new hardware or changes to the model. It is mostly a scheduling and configuration decision. In principle, it could sit on top of existing distributed training infrastructure.

I'll share results once we have the full evaluation done.

References

1. D. Narayanan et al. "Efficient large-scale language model training on GPU clusters using Megatron-LM," 2021. arXiv:2104.04473
2. Z. Li et al. "Automatically planning optimal parallel strategy for large language models," 2024. arXiv:2501.00254
3. X. Liu et al. "Galvatron: An automatic distributed system for efficient foundation model training," 2025. arXiv:2504.21411
4. D. Patterson et al. "Carbon emissions and large neural network training," 2021. arXiv:2104.10350
5. A. Lacoste et al. "Quantifying the carbon emissions of machine learning," 2019. arXiv:1910.09700
6. L. F. W. Anthony, B. Kanding, and R. Selvan. "Carbontracker: Tracking and predicting the carbon footprint of training deep learning models," 2020. arXiv:2007.03051
7. D. Gu, Y. Zhao, P. Sun, X. Jin, and X. Liu. "GreenFlow: A carbon-efficient scheduler for deep learning workloads." IEEE Transactions on Parallel and Distributed Systems, 36(2):168-184, 2025.
8. Q. Chen et al. "InternEvo: Efficient long-sequence large language model training via hybrid parallelism and redundant sharding," 2024. arXiv:2401.09149