DeepSeek’s First Big Move of 2026: A Formula for Training Giant AI Models Cheaply

In the very first days of 2026, a technical paper from DeepSeek reignited the “giant model training” debate. In the paper, which carries the signature of founder Liang Wenfeng and was prepared with a large research team, DeepSeek announces an architecture it calls Manifold-Constrained Hyper-Connections (mHC). The claim is simple but its impact is big: keeping training cost and compute load “almost constant” while scaling the model up, and at the same time reducing training instabilities at large scale.

Claims like this, especially at a time when GPU scarcity is still on the agenda and training costs have become the line item that “sets the budget,” are not just a technical development; they are also a strategic move. Because the race in foundation models is no longer only about “who is bigger?” but also about “who scales more efficiently?”

Why is this so critical now?

As large language models (LLMs) and multimodal systems grow, training gets harder in three areas:

• Compute cost: The FLOP requirement per training step increases; parallelization and data/layer distribution become more complex.
• Memory cost: Activations, optimizer states and intermediate tensors create bottlenecks, especially when targeting long context and large batches.
• Training stability: As depth increases, how the signal flows between layers becomes more critical; even small imbalances can blow up in large-scale training.

That’s why any idea that makes “signal flow” more balanced at the architecture level matters not only for performance but also for the cost/success ratio.

The ResNet legacy: It all started with residual connections

In most modern deep networks, especially the transformer family, “residual” connections are one of the core ideas that make training possible. Put simply, residual connections ease gradient flow by carrying a layer’s output “directly” to the next stage. This reduces classic problems like “vanishing gradients” in very deep networks.

But as scale grows, residual connections have a limit too: each layer largely “talks to its own neighbor.” If you want a richer flow of information, you need to widen the interaction between layers. This is exactly where the Hyper-Connections idea comes in.

Hyper-Connections (HC): “Richer” shortcuts between layers

The HC approach takes the idea of the residual connection a step further and aims to build more flexible connection patterns between layers. In the classic residual structure, there is roughly this: “Layer output + previous representation.” In HC, the signal a layer receives can be a mix of components coming not just from the previous layer but from multiple layers.

In theory, this brings two advantages:

• Richer information flow: The representation formed in lower layers is carried to upper layers through more varied paths.
• Greater expressive power: The model gains a more flexible learning space for “how much to use each layer’s information.”

But there is a risk here as well: as connection freedom increases, signals growing/shrinking and the balance breaking during training can become more likely. Especially at large scales, these kinds of imbalances can turn from “occasional” into “systematic” problems.

The core idea of mHC: Keep the freedom, but constrain it geometrically

DeepSeek’s mHC approach proposes taking the rich connection idea HC offers and making it more controlled with a “constraint.” The main goal of the constraint here looks like this:

Instead of leaving the mixing coefficients between layers completely free, keep them on a specific mathematical space (manifold) so that the signal is carried in a balanced way.

In its simplest form, the word “manifold” can be thought of as “a surface/space with certain rules, where parameters don’t wander anywhere at random.” According to DeepSeek’s claim, this constraint reduces some of the instabilities seen during training and makes the system behave “more predictably” as scale grows.

Another critical point that appears in popular summaries of the paper is this: the connection/mixing matrices carry certain properties (for example, “doubly stochastic”-like conditions). Without going deep into the technical details, constraints like these aim to prevent weights from “piling up in one direction” or “some paths becoming overly dominant.” The result: a more balanced flow.

What does the “compute load doesn’t increase” claim mean?

The critical question with architecture proposals like this is: there’s a nice idea, but in practice how much extra cost does it bring? The claim mHC makes is that it enriches connections without dramatically increasing “compute.”

This is usually possible in two ways:

• Parameter growth stays limited: The added mechanism relies not on huge matrix multiplications but on smaller-scale weighting/mixing steps.
• It’s carried by engineering optimizations: The extra load is “hidden” or minimized through practical methods like custom kernels, activation recomputation and pipeline parallelism.

Of course, there’s an important note here: the sentence “compute load doesn’t increase” usually doesn’t mean “the FLOPs are the same in theory”; in practice it means “it can be applied without meaningfully increasing total training cost.” That’s why mHC’s real impact will only become clear when it’s reproduced across different hardware/distribution configurations.

Experiments: What do the 3B / 9B / 27B scales show?

The DeepSeek team reports that it tested mHC at 3B, 9B and 27B parameter scales and got more stable results in large-scale training. Scales like these are too large to count as “toy experiments,” but they are also below the level of “the biggest frontier models.” So the right conclusion to draw is this:

mHC may be promising at a meaningful size; but its real test will come at much larger scales and in much longer training runs.

Still, gaining training stability at a scale like 27B is important because it shows the architectural idea can carry over “at least up to a certain level.”

Who could this approach be a game changer for?

mHC-like architectural improvements are especially valuable for these profiles:

• GPU/compute-constrained teams: Those who want to try a bigger or better model with the same budget.
• Efficiency-focused labs: Those who want to reduce repeated attempts by making training “stable” rather than growing model scale.
• Product teams: When uncertainty in training cost (a run crashing, instability, restarts) decreases, product planning becomes more predictable.

In short, this is attractive not only for “a better score” but also for less risky training.

Question marks: Every efficiency promise comes at a price

Every new architectural idea comes with some open questions. For mHC, a few headings stand out:

• Implementation complexity: How is the “connection constraint” applied in practice? How dependent is it on training code and kernel optimizations?
• Generalization: Does the same advantage hold across different data mixes, different tokenization, different optimizer settings and different architectural variants?
• Inference impact: While gaining stability in training, does it create latency/memory effects on the inference side?
• Ablation transparency: How much of the gains come from “the mHC idea” and how much from “engineering optimization”?

The answers to these questions will become clear through follow-up replications and open-source implementations rather than a single paper.

Why is the CEO’s signature seen as a “signal”?

One detail that draws attention in the industry is that technical announcements like this are associated with the very top names themselves. This can usually mean two things:

• The company is positioning the approach as its “core strategy.”
• It is laying the groundwork for a new model/product announcement in the near term.

Given that DeepSeek has made “headline-grabbing” model releases close to holiday periods in the past, it’s no surprise that an expectation of a new model launch in the first quarter of 2026 is forming. mHC is also being read as an “infrastructure move” feeding this expectation.

Conclusion: In 2026, the name of the competition is “scale + efficiency”

mHC’s claim fits an increasingly valuable place in the world of giant models: Being bigger isn’t enough; you need to grow more efficiently. If mHC can really increase stability and push costs down, it could start a new architectural direction not only for DeepSeek but for “frontier model training” in general.

In the short term, the clearest signals to watch will be: the reproducibility of mHC at different scales, adoption by the open-source ecosystem, and whether DeepSeek makes this approach “standard” in its next model.

Short glossary (for a quick read)

• Residual connection: A shortcut that carries the signal directly between layers.
• Hyper-Connections (HC): An approach that aims to enrich information flow by widening the connections between layers.
• Manifold constraint: A mathematical constraint that keeps parameters within a space limited by certain rules.
• Training stability: Losses/gradients not behaving “uncontrollably” in large-scale training, and the run completing reliably.