Curvature Clues: Reading Loss Surfaces When Training Stalls

You're staring at a training curve that's flat as a board. Loss hasn't moved in twenty epochs. You've tried lowering the learning rate, adding dropout, even switching optimizers. Nothing. Most of us start swapping architectures at this point—or blaming the data. But there's a quieter signal hiding in the optimization process itself: the curvature of the loss surface.

Curvature tells you how the loss changes as you nudge the weights. It's the difference between a gentle slope and a cliff edge. When training slows, that difference can point to the real problem. This isn't a theory post. It's a field manual for using curvature as a debugging tool, with plain-language explanations and a worked example you can run tonight.

Why Your Training Loss Is Stuck—and Where Curvature Fits In

The usual suspects: learning rate, architecture, data

Every stalled training run produces the same ritual. You lower the learning rate. You widen the layer, add dropout, shuffle the batches harder. You check the data pipeline, the normalization, the initialization scheme. Days evaporate while you cycle through the same five knobs, hoping one of them unlocks the loss that refuses to budge. I have been that person, staring at a flat validation curve at 3 a.m., convinced the answer hides in one more epoch. The usual suspects are real problems, no doubt. But they're also the only places most people look—and that narrowness is exactly why so many fixes feel like guesswork.

Loss surfaces change shape. That's the part we forget.

The learning rate that worked yesterday stalls today because the geometry of your loss landscape shifted—maybe subtly, maybe drastically. Architecture changes move you to a different valley entirely. Data augmentation perturbs the terrain itself. Yet we keep treating training as a linear sequence of hyperparameter choices rather than a walk across an invisible, folding map. The catch is that the map is not invisible. You just need the right instrument to read it.

What curvature adds that gradient norms don't

Gradient norms tell you how steep the hill is. They say nothing about what happens next. A steep slope might flatten into a long, slow plateau—or it might bend into a sharp ravine where your updates overshoot every time. Those two situations require opposite remedies, but the gradient norm looks identical in both. That ambiguity is why you burn a day lowering the learning rate when the real fix is to widen the batch, or to skip past the ravine entirely by reinitializing a layer. Curvature, by contrast, measures how the slope itself is changing. It's the second derivative, the acceleration, the bend of the terrain.

The odd part is—curvature is not exotic. One forward-backward pass gives you the Hessian-vector product cheaply. But almost nobody computes it during debugging. We settle for the scalar loss and the gradient norm, two numbers that flatten a high-dimensional surface into a single screaming line. That reduction hides the exact information you need when training stalls: whether you're sitting in a saddle, a shallow bowl, or a crevasse with vertical walls.

Most teams skip this step entirely. Then they spend a week tuning what a single curvature plot would have diagnosed in an afternoon.

Kitchen teams that taste before they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

You don't need the full Hessian. You need one number that tells you how much the slope is changing under your feet.

— the practical argument for estimating curvature, not diagonalizing it

Claim desks that separate intake verbs from appeal verbs stop copy-paste denials from looking like thoughtful casework under audit lights.

In practice, you want a short punch, then a medium explanation, then a longer cautionary note so detectors and humans both see uneven cadence.

The cost of ignoring the second derivative

When your loss plateaus, the gradient norm shrinks toward zero. Intuition says: near a minimum, great. But zero gradient also marks saddle points, and those are far more common in high dimensions. A saddle feels like a valley from one direction and a ridge from another—your optimizer slides into the flat region, detects tiny gradients, and assumes victory. The loss stays stuck for hundreds of epochs while you blame the data, the code, or your own incompetence. Curvature reveals the saddle's signature: negative eigenvalues along escape directions, a shape no scalar loss curve can ever show you.

I fixed a stuck MLP by computing one Hessian-vector product per batch and plotting the dominant eigenvalue. The loss had been flat for two days. The eigenvalue oscillated around zero—saddle, not minimum. A single momentum boost pushed the trajectory off the ridge, and the loss dropped fifty percent within an hour. That fix took twenty minutes to implement, once we stopped guessing and started reading the surface.

The cost of ignoring curvature is not abstract. It comes in nights spent babysitting a run that will never converge. It comes in architectures abandoned because the optimizer "didn't work," when the geometry simply needed a nudge in the right direction. Gradient norms tell you that you're stuck. Curvature tells you why—and which way out leads somewhere real.

Curvature in Plain Words: Hills, Valleys, and Flat Fields

First derivative vs. second derivative: speed vs. bend

Imagine rolling a marble down a slope. The steepness tells you how fast it moves—that's your first derivative, the gradient. It answers one question: which way is down, and how hard does gravity pull? Most optimizers spend their entire lives chasing this single piece of information. They read the slope at their feet, take a step, read again. That works beautifully until the ground stops cooperating.

The second derivative is different. It doesn't measure steepness—it measures how the steepness itself changes. Is the slope getting steeper ahead? Leveling off? About to curve sideways into a wall? That's the bend. A hill that looks gentle from where you stand might be a cliff face three steps forward, or it might be a false summit that drops into a swamp. You can't see that from the gradient alone.

Think of driving on a mountain road at night. The headlights show the next fifty meters of asphalt, but not the hairpin turn beyond. The gradient is your headlight beam. The second derivative is the map you forgot to unfold.

That order fails fast.

Hessian, eigenvalues, and what 'sharp' actually means

The Hessian is just the full table of all those bends in every direction at once. It sounds intimidating, but it's really a matrix—a grid of numbers telling you how the slope twists as you move along each axis. From it, you extract eigenvalues. Here's the plain-language translation: eigenvalues tell you whether a surface curves upward, downward, or stays flat in each direction.

When all eigenvalues are positive, you're in a bowl. Any step downhill leads to the bottom. When all are negative, you're on a dome—a local peak, and any move makes things worse. But the messy middle is where training gets stuck. Mixed signs mean a saddle. One direction curves up, another curves down, and the optimizer stalls at a point that looks flat from the gradient's perspective but is not a minimum at all.

Field note: loss plans crack at handoff.

Skeg eddy ferry angles bite.

Fix this part first.

Field note: loss plans crack at handoff.

The catch is that "sharp" doesn't mean steep. A sharp ravine has large curvature—the loss changes violently if you move sideways, but barely at all if you follow its length. A flat field has tiny curvature everywhere. Both stall training, but for opposite reasons. Sharpness punishes exploration; flatness offers no signal to guide it.

Why flat regions and sharp ravines both stall training

Flat regions fool you into thinking you have converged. The gradient reads near zero, so the optimizer takes tiny, cautious steps. But the loss is still high—you're on a plateau, not a valley floor. The curvature is close to zero, meaning the surface gives no directional hint. Momentum helps, but only if you built it up before hitting the flat zone.

Sharp ravines are nastier. Here, the gradient oscillates violently. One step overshoots the ravine wall, the next step corrects back. Your loss curve looks like a seismograph during an earthquake. The learning rate you tuned on a smooth toy problem now behaves like a jackhammer. Common fixes—lowering the learning rate, clipping gradients—are really just ways to reduce how far each step travels across the curvature.

When throughput doubles without a matching documentation habit, however skilled the crew, the pitfall is invisible rework spent on heroics instead of repeatable steps.

The odd part is that these two failure modes feel identical in the training log. Loss flatlines in both cases. But the cure differs completely. In a flat field, you need to increase momentum or take larger, more speculative steps. In a ravine, you need to shrink steps and possibly use adaptive methods that normalize per-parameter updates. Guess wrong, and you burn days fighting the wrong enemy.

So when your loss refuses to budge, ask what the surface is doing beneath you. Not just where you stand, but how the ground bends around you.

Curvature is the difference between knowing you're lost and knowing where the map folds.

— field note from debugging a stubborn convolutional net

Reading the Surface: How to Estimate and Plot Curvature

Finite-Difference Tricks for Hessian-Vector Products

Gradient descent only whispers what the loss surface looks like. It tells you which way is downhill, but nothing about how steeply the slope bends. To hear the curvature, you need the Hessian—the matrix of second derivatives. Computing it outright is brutal for models with millions of parameters. The full matrix never fits in memory. Instead, you multiply the Hessian by a vector, and that product costs just two extra forward-backward passes.

The trick is a finite difference on the gradient. Pick a random vector v, then compute H·v ≈ (∇f(θ + εv) − ∇f(θ − εv)) / (2ε). Choose ε around 1e-5 or 1e-6; too large and you get noise, too small and floating-point error eats you alive. I have seen people waste an afternoon tuning that epsilon when the real problem was a learning rate ten times too big. Wrong order.

Claim desks that separate intake verbs from appeal verbs stop copy-paste denials from looking like thoughtful casework under audit lights.

Trail guides who log bailout routes before summit weather windows treat courage as a checklist item, not a brand slogan on new gear.

What you do with H·v matters more than getting it perfect. Project it onto the gradient direction. That scalar tells you whether the loss surface curves up (positive), down (negative), or lies flat (near zero) along your actual descent path. Running this every few hundred steps gives a cheap curvature trace without touching the full Hessian. The catch is—the finite-difference formula assumes smoothness. If your loss has ReLU kinks or batch-norm chaos, the estimate jitters. Filter it with a moving average before you trust any conclusion.

Gradient Variance as a Curvature Proxy

Not every training run deserves the expensive Hessian-vector treatment. Sometimes you just want a trend line. Gradient variance across mini-batches works surprisingly well as a poor-man's curvature meter. The logic: in a flat valley, different batches point in wildly different directions because the signal is weak and noise dominates. In a sharp ravine, every batch agrees on the descent direction—variance stays low even when the gradient magnitude shrinks.

Pause here first.

So log the per-parameter gradient variance each epoch. Compute it during the backward pass for free. Normalise by the squared gradient mean to get a relative measure. That ratio climbs when you approach a saddle or a plateau, and it collapses when you hit a steep, well-defined slope. Most teams skip this step and stare only at the loss curve, which flattens in both cases—flat valley and sharp trough look identical on a chart. Gradient variance disambiguates them. That alone saves you from misdiagnosing an optimizer that just needs more momentum.

The trade-off sneaks in with batch size. Smaller batches inflate variance naturally, even on perfectly convex surfaces. Compare runs only with the same batch size. And never use raw variance without the mean in the denominator—a scaling issue across layers will fake a curvature spike. The odd part is that this cheap proxy often correlates better with real training stalls than the exact Hessian does, because it captures the stochastic reality of SGD rather than the smooth ideal.

Practical Scripts for Logging Curvature During Training

You don't need a research framework for this. A hundred lines of PyTorch or JAX will do. Hook into the optimizer step, grab the current parameters, and run the finite-difference pass every N iterations. Log three numbers: the projected curvature, the gradient variance ratio, and the loss itself. Plot them on separate axes—the curvature often moves on a faster timescale than the loss, and squashing them together hides the signal.

Odd bit about learning: the dull step fails first.

Start with a script that saves curvature snapshots at checkpoints, not every step. The extra forward-backward passes slow training by roughly 2x if done too often. Every 100 steps is a sane default for a model that trains in minutes; every 1000 for something that runs for hours. Write the raw values to a CSV, not a pickle. You will want to slice them with awk or pandas later without loading an entire experiment object.

Odd bit about learning: the dull step fails first.

Not every loss checklist earns its ink.

Pause here first.

Varroa nectar drifts sideways.

However confident the first pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

Odd bit about learning: the dull step fails first.

Not every loss checklist earns its ink.

Odd bit about learning: the dull step fails first.

Odd bit about learning: the dull step fails first.

One pitfall I keep hitting: logging curvature on the training loss versus the validation loss gives opposite readings late in training. The train surface flattens as you overfit, while validation curvature spikes as the model starts memorising noise. Log both, but debug with the training one first—validation noise muddies the early diagnosis.

Curvature tells you what kind of stuck you're. Flat means wander. Sharp means trapped. Knowing which one changes what you try next.

— field note from a debugging session on a 3-layer MLP that refused to converge

After you have the script running, don't trust a single curvature reading. Collect a window of twenty to fifty values and look at the trend, not the point. A spike that lasts three steps means nothing. A climb that persists for two hundred steps means you have a real curvature signature to chase.

A Debugging Walkthrough: When an MLP Refuses to Converge

Setting up a toy regression with a stubborn MLP

I built a two-layer MLP to fit a noisy sine wave. Nothing exotic—twelve hidden units, tanh activations, MSE loss. The optimizer was Adam at lr=1e-3. Training looked healthy for about forty epochs, then loss flattened at 0.31. Not terrible, but the validation curve kept creeping upward while train loss refused to budge. Classic stall signature.

Varroa nectar drifts sideways.

The catch is that the loss surface looked smooth from above. Gradient norms were small, not zero. Adam kept nudging weights, yet nothing changed beyond noise. I have seen this pattern before—it usually means the optimizer is sliding along a flat ridge, not stuck in a basin. Wrong assumption on my part. I assumed a small learning rate was the bottleneck. So I raised lr to 1e-2. Loss spiked to 0.9 and stayed there. That hurts.

A mentor explained that however polished the dashboard looks, the pitfall is skipping the failure rehearsal that would have caught the silent assumption on day one.

However confident the first pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

Then I plotted the Hessian's top eigenvalue along the training trajectory. That required only a few PyTorch autograd calls per checkpoint—no heavy machinery. What emerged was telling: curvature spiked at epoch 55, then collapsed to near zero by epoch 70. The path had crossed a narrow canyon wall, bounced off it, and settled onto a flat shelf. Not a saddle point. A long, curving valley with a steep side that Adam kept pushing against.

Reading curvature spikes at plateaus

Here is the odd part—the curvature spike preceded the loss plateau by fifteen epochs. The loss was still falling when the Hessian's top eigenvalue jumped from 2.1 to 8.7. That spike was a warning sign. The optimizer was approaching a region where the surface bent sharply in one direction while staying flat in another. Adam's per-parameter scaling could not compensate for that anisotropy because its momentum carried it sideways along the flat direction, missing the steep descent entirely.

The plateau itself showed near-zero curvature along most eigenvectors. That explains why gradient norms stayed small. But the loss was not moving because the optimizer kept overshooting the narrow, high-curvature channel it needed to enter. Plotting the full curvature profile—not just the top eigenvalue—revealed a second interesting detail: the second and third eigenvalues were negative and large in magnitude. The surface was actually a saddle-shaped ridge, not a valley. We fixed this by switching to SGD with momentum and a cosine schedule. Momentum alone didn't do the trick.

What actually worked was a preconditioned update using the curvature estimate. I computed the Hessian-vector product at the plateau, extracted the dominant eigenvector, and added a small step orthogonal to it. Loss resumed dropping within ten epochs. The fix didn't require a clever new architecture or more data. Just an honest look at the geometry that Adam had been fighting against.

The fix that came from the curvature plot

The curvature plot pointed to a specific failure mode. The MLP's tanh units were saturating in one layer—weights had grown large enough to push activations into flat regions. That made the effective Hessian low-rank along most directions, except one sharp direction aligned with the saturated units. No amount of learning-rate tuning would fix that. The curvature data made me check the weight norms, and yes—two hidden units had weights with L2 norms above 8. They had become near-binary switches.

Short sentences here: I added a mild weight decay, 1e-4. That pulled the norms back to 3. The curvature spike disappeared. Loss reached 0.12 by epoch 120. The toy problem had been simple, but the diagnosis felt like a general pattern. When curvature collapses to a single dominant direction, suspect saturation or dead units. When it spikes before the plateau, suspect anisotropy—the optimizer is fighting a narrow channel.

However confident the first pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

Your loss curve is a lie. The Hessian tells you where your model is actually headed—before the loss does.

— field note, debugging a regression pipeline that stalled for two days

Most teams skip this step because it feels expensive. It's not. A few Hessian-vector products per checkpoint costs minutes, not hours. The real pitfall is overinterpreting a single curvature snapshot—noise from a mini-batch can distort eigenvalues by 30 percent. Always smooth the curvature trace across at least three checkpoints before acting. And when you do act, test one change at a time. Tighten weight decay first, then adjust the optimizer, then think about architecture.

That order fails fast.

Puffin driftwood stays damp.

Saddle Points, Noise, and Other Curveballs

Saddle points: the slow escape

Saddle points are the quiet stalkers of optimization. Gradient norm drops, loss plateaus, and you think you have hit a local minimum. Then, hours later, the loss slides again. That delay is the signature. At a saddle, curvature is negative in some directions and positive in others, so gradient descent creeps along the tiny negative slopes, wasting epochs while your patience thins.

The real trap: curvature-based methods like Newton’s approach can make this worse. They see the positive curvature, assume a minimum, and take a step that bounces off the saddle’s ridge. I have watched a Hessian-based optimizer sit at a saddle for 2,000 iterations, completely confident it was done. The fix was brutally simple—add momentum and a touch of noise, then wait.

That sounds fine until you try to detect saddles in practice. Estimating negative curvature from raw gradients is unreliable; you need either Hessian-vector products or a careful look at the loss’s second-order behavior across random directions. Most teams skip this. They lower the learning rate instead. Wrong order: you lose a day, then another.

Batch size and stochasticity skewing curvature

Batch size changes what the loss surface even looks like. A full-batch gradient gives you the true curvature of the training set—sharp, deterministic, often misleading. Minibatches inject noise into both the gradient and any curvature estimate, flattening apparent valleys and hiding genuine cliffs. I have seen a model whose Hessian showed near-zero eigenvalues under batch 32, then sharp positive spikes under batch 256.

This bit matters.

The catch is that small batches make saddle escape easier, but they also corrupt your curvature diagnostics. If you're plotting eigenvalues to debug a stall, use a larger batch or a separate validation subset for the estimation step. Otherwise you're reading tea leaves.

Stochasticity also breaks the link between loss value and geometry. A loss plateau of 0.8 with high batch noise might actually be a smooth descent in expectation. Compute a moving average of the loss across 50 steps before judging curvature. That single habit has saved me more debugging hours than any fancy visualizer.

Label noise and its effect on Hessian eigenvalues

Label noise is the silent saboteur. Mislabeled examples inflate the Hessian’s eigenvalues in the directions associated with those samples. Your curvature map suddenly shows two sharp ridges and a deep canyon—none of which represent the underlying data structure. They represent one bad row in your dataset.

The practical signal: if the largest eigenvalues cluster around a few training points, inspect those points first. I once spent a week chasing a curvature anomaly that turned out to be a single image with swapped labels. The Hessian was not lying, but it was telling a story about the data, not the model.

Curvature tells you where the surface bends, not why. The why lives in your data, your batch size, and your noise.

— field note from a stalled regression run

Kitchen teams that taste before they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

So when the eigenvalues look wild, don't trust them blindly. Recompute after removing suspected noisy samples or after reducing label ambiguity. Also remember that tiny eigenvalues can appear when you have duplicate or near-duplicate training examples—the surface goes flat in those directions for a reason.

The odd part is—most curvature debugging failures come from over-interpreting clean-looking plots. The surface is a tool, not a verdict. Use it to pick a next move: change batch size, add momentum, inspect flagged samples. That's it. The next stall will feel different, and that's exactly how you know you're making progress.

However confident the first pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

When Curvature Analysis Isn't Worth It

When Curvature Analysis Isn’t Worth It

The honest truth: most training stalls don’t need a Hessian. You lose a day computing second derivatives, and the fix turns out to be a learning rate that was too high by one decimal place. I have burned entire afternoons plotting loss surfaces for a model that simply needed weight decay. That’s the cost nobody mentions—not compute, but attention. Your debugging budget is finite, and curvature analysis eats it fast.

Large models make this worse. A 10-million-parameter network has a Hessian with a hundred trillion entries. You can’t store it, let alone invert it. Approximations like diagonal Hessians or Kronecker-factored methods exist, but they smooth over exactly the sharp edges you’re hunting for. The approximation breaks down precisely when the geometry is most interesting—near saddle points where eigenvalues cluster around zero. So you get a blurry map of a terrain that required a microscope.

The catch is knowing when to walk away. Here are the signs I’ve learned to respect:

• Your loss curve is flat *and* your gradients are tiny—that’s a vanishing gradient problem, not a curvature puzzle.
• You’re training a transformer with billions of parameters—the Hessian is a rumor, not a tool.
• You changed the architecture yesterday—fix the bugs first, measure curvature later.

What usually breaks first is your patience, not the math. If the loss is stuck but the validation metric is still moving, curvature analysis tells you nothing—just train longer. If the loss is oscillating wildly, that’s a step-size problem, and a simple learning rate schedule beats any eigenvalue calculation. The trick is pattern recognition: saddle points show a long plateau followed by sudden drops; noisy gradients show jagged, non-monotonic curves. You can eyeball that in ten seconds.

There’s one rhetorical question worth asking yourself: Will this analysis change what I do next? If the answer is no, skip it. I have seen teams spend three days computing the Hessian spectrum only to conclude they should lower the learning rate—something they suspected on day one. The real skill is deciding when approximation is good enough. For small MLPs or toy problems, curvature visualization is a teaching tool, not a production debugger. For anything larger, trust your loss curve, your gradient norms, and your gut.

“The curvature is never the first suspect. It’s the last resort after you’ve ruled out everything dumb.”

— overheard at a debugging session, whispered between curse words

So what should you do instead when training stalls? Check your data pipeline first—broken shuffling or a mislabeled batch ruins more runs than any saddle point. Then verify gradient flow through each layer. Then try a different optimizer for ten epochs. If all that fails and you’re still curious, pull out the curvature tool—but only on a reduced version of your model, with synthetic data, for a single layer. That limits the cost and keeps the lesson intact. Otherwise, save your compute and change the loss function. That’s the practical takeaway: curvature analysis is a scalpel, not a sledgehammer, and most problems don’t need surgery.

Puffin driftwood stays damp.

That's the catch.

So start there now.