What to Fix First in a Chaotic Loss Surface: Learning Rate or Initialization?

You've seen the plot: loss bouncing like a pinball, never settling. Every knob screams for attention—batch size, momentum, weight decay. But two stand out: learning rate and initialization. You have finite GPU hours. Which do you twist first?

We're not here to sell you a magic formula. Instead, we'll walk through the decision, compare criteria, and show you the trade-offs. By the end, you'll know exactly which lever to pull for your specific mess.

The Decision: Who Must Choose and Why Now

When chaotic loss surfaces appear

You're staring at a training curve that looks like a seismograph during an earthquake. Loss spikes. Plateaus that last for hours. Then a sudden drop that you can't reproduce. This is not a toy MNIST experiment. This is a production model—maybe a transformer variant, maybe a deep convnet—and the surface is genuinely hostile. I have seen teams burn two weeks here, tweaking one knob, then another, never sure which move actually mattered. The chaotic loss surface doesn't announce its arrival. It creeps in around epoch three, when the learning rate feels slightly off and the weights seem to have landed in a bad neighborhood. Most engineers freeze. They reach for the learning rate first. That's a reflex—and reflexes can cost you a Monday.

The wrong starting point compounds itself.

Here is the concrete scenario: your validation loss stopped improving at 1.8, and now it wobbles. The gradient norms are not exploding, but they flicker. You try reducing the learning rate by half. The wobble smooths out—good—but the loss flatlines at 2.1. Worse than before. So you crank the learning rate back up and try a different initialization scheme. That bounces you to a new region of the surface, and suddenly the loss drops to 1.6. For two epochs, then it shatters. The catch is: you have no idea whether the learning rate or the initialization caused the drop. They interact. They fight. Getting the order wrong means you spend a week debugging interactions instead of converging.

‘Most people assume the learning rate is the primary dial because it's easier to reach. That assumption has killed more runs than a bad batch size ever did.’

— overheard during a debugging session at a mid-size ML shop, two days before a deadline

Stakes of getting the order wrong

Order matters because a chaotic surface punishes sequential reasoning. Fix the learning rate first? You might smooth gradients that were never the real problem. Fix initialization first? You might land in a basin that needs a higher learning rate than you dare to set. The practical cost is not just time—it's trust. After three failed tuning rounds, your team starts doubting the architecture, the data pipeline, even the optimizer. Doubt slows everything. I have watched a senior engineer swap from Adam to SGD, then to Ranger, then back to Adam, all because the initial choice of learning rate masked a terrible initialization. That took six hours. A simple check—plotting the loss surface around the starting point—would have revealed the issue in twenty minutes. Most teams skip this. Wrong order.

Not yet.

The time pressure in real projects makes it worse. You have a demo on Friday. Today is Tuesday. The loss is thrashing. Your instinct says: change something, anything. That's the moment when the decision calcifies. Pick learning rate arbitrarily, and you cement a bad trajectory. Pick initialization arbitrarily, and you commit to a region the optimizer can't escape. The messy truth is that neither fix works in isolation if the surface is truly wild. You need a diagnostic step before you touch any knob.

Option Landscape: Three Ways to Tame the Chaos

Learning rate scheduling — because the surface changes under your feet

A fixed learning rate on a chaotic loss surface is like driving a rally car with the throttle taped down. The early slopes might be gentle, so a moderate rate works fine — then you hit a ravine and the optimizer launches into orbit. I have debugged enough of these blowups to know the fix is rarely a smaller global rate; it's a schedule that adapts as the geometry shifts. Step decay, cosine annealing, and cyclical schedules each buy you something different. Step decay gives the optimizer room to settle into a basin, then tightens the exploration window. Cosine annealing cycles the rate up and down, which can bounce the weights out of sharp minima into flatter ones. The catch is picking the right schedule family without a grid search that costs a week of GPU time. Most teams skip this: they slap on ReduceLROnPlateau and assume the plateau detector knows what a plateau is. It doesn't — not on a chaotic surface. The valley you think is a plateau might be a temporary flat spot before a cliff.

Wrong schedule often looks like the loss stuttering sideways for fifty epochs.

Cyclical schedules bring another pitfall: they can re-introduce the chaos they were meant to tame. If the cycle amplitude is too wide, the weights get thrown back into a high-loss region every restart. That hurts. The trade-off is clear — narrow cycles stabilize training but may not escape bad basins; wide cycles escape but risk erasing learned features. I have seen ResNet-50 on CIFAR-10 lose 4% accuracy simply because the cosine cycle bottomed at a rate that was still too aggressive for the batch-normalization layers. The fix was a warm restart with a lower max rate. Not glamorous. Effective.

Initialization strategies — the quiet variable nobody tunes

Learning rate gets all the attention. Initialization gets the blame when nothing else works. Xavier, He, orthogonal, and their variants each assume a specific propagation dynamic. Xavier (Glorot) works well for tanh activations because it keeps variances balanced through symmetric non-linearities. He initialization was designed for ReLU — it accounts for the dead half of the activation and doubles the gain. Orthogonal initialization, meanwhile, preserves gradient norms across very deep networks. The odd part is: most practitioners use He for everything and call it done. That works until your loss landscape has a narrow ravine aligned 45 degrees off the weight axes. Then the singular values from He initialization can point the gradient in a direction that oscillates rather than descends.

‘We spent two days tuning the learning rate only to find the initial variance was wrong by a factor of two.’

— engineer at a medical-imaging startup, after switching from He to orthogonal for a 50-layer segmentation network

What usually breaks first with poor initialization is not the loss value — it's the gradient norm. Exploding norms signal that the initial weights were too large for the activation function's saturation regime. Vanishing norms mean the opposite: the signal died before reaching the output. Both look like the loss is stuck, but the fix is completely different. For exploding gradients you clip or shrink initialization scale; for vanishing gradients you increase the scale or switch to leaky ReLU. The trap is thinking that batch normalization will rescue a bad initialization. Batch norm compensates for scale shifts, not structural misalignment between the weight distribution and the loss geometry. I have watched teams normalize their way into a false sense of stability, then hit a gradient explosion three hundred epochs later when the moving averages drifted.

Hybrid approaches — when neither alone is enough

The most stubborn landscapes require both levers pulled simultaneously. A sensible pattern: initialize with He or orthogonal based on depth, then pair it with a warmup schedule that starts the learning rate near zero and ramps over 5–10 epochs. The warmup lets the optimizer find a coherent gradient direction before the rate becomes aggressive. That sequence alone has saved more training runs than any single hyperparameter optimization I have seen. Another hybrid trick is to initialize the last layer's weights to zero — not small, but exactly zero — so the network starts from a trivial solution and the early gradients are driven entirely by the residual signal. This works surprisingly well for deep residual architectures where the initial loss plateau masks what the learning rate should be.

The risk with hybrids is complexity creep. Two knobs become four when you add schedule shape and initialization variance scaling. The fix is to constrain the search: fix the initialization strategy first using a short proxy run of 20–30 epochs with a conservative learning rate. Once the gradient norms remain stable across that window, then tune the schedule. Do it in the reverse order — tune schedule first on a bad initialization — and every adjustment compensates for a problem that shouldn't exist. The messy truth: you will debug both eventually, but the order determines whether you spend an afternoon or a week. Start with the weights, then the throttle. That order has never failed me. The reverse has failed repeatedly.

How to Compare: Criteria That Actually Matter

Stability of early training

Watch the first twenty steps. That's where most breakdowns announce themselves. A bad initialisation sends gradients skyward inside three batches—loss jumps from 2.1 to 47 in a single update, then oscillates wildly. Wrong learning rate, by contrast, produces a quieter failure: loss declines for eight steps, flatlines, and never recovers. I have seen teams waste two weeks tweaking the LR schedule when the real culprit was a weight distribution that ignored layer depth. The metric here is simple: measure the variance of the loss gradient norm over the first 200 iterations. If it exceeds 0.5× the mean norm, you probably need better initialisation, not a smaller step size. The catch is that both faults can look identical on a smoothed TensorBoard curve—zoom into the raw per-step values.

Flag this for machine: shortcuts cost a day.

Flag this for machine: shortcuts cost a day.

Convergence speed vs. final loss

Speed tempts everyone. A high learning rate makes the loss plummet through the first epoch—great for demos, awful for production. What usually breaks first is the late-stage jitter: the model finds a basin, then can't settle inside it. Initialisation affects the opposite end. A lucky starting point might converge fifty percent faster, but it also might land in a narrow valley that generalises poorly. How do you decide? Run two ablations: fix the LR at 3e-4 and swap between Xavier, Kaiming, and orthogonal initialisations; then fix the initialisation and sweep LRs from 1e-4 to 1e-2. Compare the ratio of validation loss at convergence divided by steps to convergence. A ratio under 0.01 means you're trading final quality for speed—acceptable only if you plan to fine-tune later.

Most papers report the loss after 100 epochs. In practice, the first 100 steps predict eighty percent of your debugging pain.

— engineer at a robotics startup, after a three-week regression hunt caused by a 1.2 initialisation scale

Robustness to architecture depth

Depth amplifies every mistake. A 12-layer transformer might tolerate a 0.1 LR drift; a 48-layer version of the same design blows up in the middle blocks. The odd part is—initialisation errors compound exponentially, while learning rate errors compound linearly. Test this yourself: take a shallow model (3 layers) and a deep variant (18 layers). Train both with the same LR and initialisation. If the deeper model shows a sudden loss spike around layer 12 during backprop, your initialisation is the weak link. If the shallow model converges but the deep one plateaus at a higher loss, the LR is too conservative for the added depth. We fixed this once by switching to residual scaling initialisation and keeping the original LR—the loss curve went from sawtooth to smooth in one change. That said, don't trust blanket heuristics: a depth multiplier of 0.1 works for ReLU networks but kills tanh activations.

Trade-Offs: Learning Rate vs. Initialization Side by Side

When LR dominates (shallow nets, small data)

Three-layer convnet on CIFAR-10? You can initialize with anything reasonable—Glorot, He, even a naive normal distribution—and the thing will still converge, provided the learning rate is sane. I have watched teams swap init schemes across a dozen runs and see maybe 2% validation difference. Then they nudge the LR by 0.001 and the entire loss surface tilts. What is happening here is a geometry problem: shallow nets have relatively smooth, convex-ish basins. The initial position barely matters because gradients are consistent and well-conditioned. The learning rate, however, sets the step size over that bowl.

However confident the first pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

Too large and you oscillate out; too small and you crawl forever. The odd part is—small-data regimes amplify LR sensitivity further. With only a few thousand samples, the gradient estimate wobbles. A high LR turns that wobble into divergence.

Kitchen teams that taste before they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

Low LR stabilizes it. So if you're prototyping on a tiny dataset with a shallow architecture, fix the LR first. Spend zero energy on init. Wrong order here costs you epochs, not days.

One concrete example: we had a regression net, four layers wide, 2,000 training points. The init was plain uniform(-0.1, 0.1). Loss plateaued around 0.34.

Name the bottleneck aloud.

Someone tried Xavier init—same plateau. Then we halved the LR from 0.01 to 0.005.

Nebari jin moss stalls.

Loss dropped to 0.21 in 15 epochs. That hurts, because it means dozens of wasted grid searches on init that never mattered.

When init dominates (deep nets, transformers)

Now stack 20 layers. Or throw in multi-head attention with residual streams. The story flips completely. Here, initialization decides whether gradients survive the first backward pass or vanish into numerical noise. Learning rate can compensate—to a point. But I have debugged a transformer where the LR was 3e-4 (standard), and the loss never went below 8.0 after 10k steps. Every troubleshooting guide said "adjust LR." We tried 1e-4, 5e-5, 1e-3. Nada. The real culprit? The embedding scale was off by one order of magnitude—initialized with std 1.0 instead of 0.02. That single number amplified activations into every subsequent layer, and the optimizer was fighting a blown-out variance problem, not a step-size problem. Once we fixed init, the original LR worked fine.

The catch is symmetry-breaking. Deep nets need each layer's output variance to remain bounded. Transformers demand precise control over residual branch scaling—the famous "Pytorch default init" that sinks GPT-style models. A bad init produces a loss surface with almost no gradient signal. Adjusting LR then feels like turning the volume knob on a microphone that's unplugged. No effect. So for depth ≥ 10, or any architecture with skip connections and normalization layers, check init before touching LR. The seam blows out not because the step is too big, but because the starting point was already off the cliff.

“I have never seen a deep net saved by a clever LR schedule after a truly broken initialization. But I have seen the reverse happen every month.”

— conversation with a colleague who maintains an internal training infra for 50+ models; not a statistic, just pattern recognition.

The middle ground: both wrong, one less wrong

Most real projects sit in a gray zone: medium depth (8–12 layers), moderate data (50k–500k samples), one or two normalization layers. Here, neither LR nor init is obviously broken, yet the loss surface still looks chaotic. What usually breaks first is the interaction. A slightly aggressive init (say, std 0.1 for a 12-layer ReLU net) creates dead neurons early. A moderate LR then can't revive them because gradients for those units are zero. But if you raise the LR, the alive units overshoot. You're stuck: each knob fixes one symptom while aggravating another.

Odd bit about learning: the dull step fails first.

Odd bit about learning: the dull step fails first.

Tactic I have used: pick the knob with the narrower safe range. Learning rate typically has a usable window of maybe one order of magnitude (0.001 to 0.01).

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

Init scale has a window closer to half an order (0.01 to 0.03 for many deep configs). In that gray zone, fixing init first is less likely to blow up later—the safe init range is tighter, meaning a mistake there kills you faster.

Kitchen teams that taste before they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

So you tune init to a conservative value (slightly low variance, slightly high gain if using batch norm), then sweep LR. That order has saved me roughly two days of debugging per project. Not elegant. But it works.

One pitfall: people try to middle-ground by using both a mediocre init and a mediocre LR. Returns spike, stall, then neither knob has leverage. You end up with a loss curve that looks like a flatworm—thrashing, no direction. The fix is brutal: reset one of them to a safe default (He init with gain 1.0, or LR 1e-3) and tune the other aggressively. Then swap. Don't attempt simultaneous Bayesian optimization on both until you have seen each one isolated. That's the messy truth: in the middle ground, one is always less wrong than the other. You just have to be honest about which.

Implementation Path: Steps After You Choose

Tuning LR first: linear warmup, cosine decay

You pick learning rate first. Good instinct—most chaos on a loss surface comes from step size, not starting point. Here is the concrete path: set initial scale to 1e-4, then add a linear warmup over 5–10% of total steps. Why warmup? Because early gradients are noisy—your model hasn't seen enough data to know which direction is real. Without it, that first big step kicks you off a cliff. I have seen training runs collapse inside 200 iterations because LR hit 0.01 on cold weights. After warmup, switch to cosine decay scheduling—it drops the rate smoothly, not in harsh stair-steps that re-shock the optimizer.

Code hint: PyTorch? Use torch.optim.lr_scheduler.LinearLR then CosineAnnealingLR. Stick a callback that logs gradient norms every 10 batches. If you see norms spike above 10× the rolling average, your warmup is too short.

Monitor validation loss at 20%, 50%, and 80% of training. If it plateaus early and refuses to budge, your init now matters—so you switch lanes.

Fixing init first: proper scaling and residual connections

Maybe your loss surface is spiky from step one—loss starts at NaN or oscillates like a broken fan. That sounds like bad initialization. For ReLU networks: Kaiming uniform, fan_out mode. For transformers: variance scaling at 1/sqrt(d_model) on each sublayer. We fixed one Vision Transformer that kept exploding at step 500—turns out the QKV projection had init gain 2.0 instead of 1.0. One line change, loss dropped by 40% within an epoch.

The odd part is—most teams skip residual scaling. If you stack 20 layers and each one adds unit variance, the output variance grows linearly. That kills gradient flow. Add a learnable scale factor (alpha) per residual stream, or use Pre-LN instead of Post-LN. I use a simple init rule: std = gain * sqrt(2.0 / fan_in) for conv layers, then clamp first-layer weights to ±0.1. Wrong order here burns half a day.

‘We tuned LR for three days. Then I changed the init scaling—loss halved in one warmup cycle.’ — intern who saved the run

— real story from a colleague debugging a 12-layer GNN

Validation checks early

Don't wait for epoch 10. Check within 100 steps. Plot three things: gradient L2 norm, loss on a frozen validation batch, and parameter update ratio (update magnitude / parameter magnitude). If update ratio sits above 0.5, your LR is too high regardless of init. If it's below 1e-6 and loss is flat, init killed your gradients—signal is dead, weights aren't moving. That hurts more than a slow LR, because you chase ghosts for hours.

Most teams skip this: freeze half a minibatch, run forward pass twice—once with current init, once with random re-init (same seed). Compare loss variance. High variance? Init is brittle, fix it first. Low variance but high loss? LR is the bottleneck. We catch 70% of surface issues inside 50 steps this way. The catch is you need this wired before training starts—otherwise you're guessing. No guesswork. Run these checks, pick your first fix, and the next 24 hours become productive.

Risks of Choosing Wrong—or Not Choosing at All

Exploding, Vanishing, and the Gradients That Just Give Up

The most spectacular failure mode is also the easiest to detect—if you're watching. Loss spikes to infinity in three steps, or your model outputs the same logit for every input. Wrong order here: a bad learning rate amplifies bad initialization, and vice versa. I have seen people spend a week debugging a Transformer that simply refused to learn, only to discover the initial weights were sampled from a distribution two orders of magnitude too wide. The gradients didn't vanish—they detonated. A single forward pass produced activations in the millions, and backprop turned those into a wall of NaNs by epoch two. The fix was not a learning-rate schedule; it was Xavier init, applied silently, and suddenly the training ran.

Vanishing gradients are sneakier. No explosion, no crash—just a loss that flatlines at 8.47 for eleven hours.

The model is technically training, but nothing moves. Dead neurons propagate silence, and the surface flattens into a plateau that looks like a local minimum but is actually a structural dead zone. How do you catch this early? Track gradient norms per layer. If the first three layers show norms below 1e-6 while the final layer jumps around, you have a vanishing problem—and cranking the learning rate won't fix a bad initial spread. It will only make the top layer thrash while the bottom stays frozen. The catch is that most dashboards monitor only the loss curve. That's not enough.

Reality check: name the learning owner or stop.

Reality check: name the learning owner or stop.

Wasted Compute on Dead Neurons

A subtler betrayal: the model learns, but half its capacity sits idle. ReLU units that never fire again after epoch four. This is not gradient vanishing—it's gradient death. A learning rate that's too aggressive shoves activations into the negative regime, the gate slams shut, and no subsequent update ever reopens it. The odd part is—the loss keeps dropping. Slowly. Progress exists, but you're effectively training a smaller network inside the one you designed, paying full compute for partial brain. We fixed this once by cutting the learning rate in half after epoch three, but the real culprit was initialization: the weight mean was slightly negative, and the first big update pushed everything left of zero. The lesson: monitor dead-unit ratios per layer, not just loss. If more than 15% of neurons in a layer are permanently off, your initialization bias is wrong, and no learning-rate scheduler will revive them.

‘A dead neuron is not a pruned neuron. It's a silent tax on your GPU.’

— overheard at a debugging session, after watching a ResNet-50 waste 60% of its filters on a Vision task

That hurts because it's invisible in the validation curve. You only see it when you dump activation histograms. Most teams skip this until week three of stalled progress.

False Convergence Plateaus

The loss flattens. The validation metric stops improving. You declare victory, export the weights, and deploy. Then production reveals a 12% accuracy gap. This is false convergence—the model settled into a shallow basin that the training loss considered acceptable, but the surface near initialization was so chaotic that gradient descent never escaped the starting region. A high learning rate can blast through bad initial zones, but if your initialization places you in a narrow valley, a high LR will just bounce you out repeatedly, never settling. A low LR will crawl down the valley and stop, convinced it reached a minimum. Neither is right. The real trick is detecting this early: plot loss versus a random projection of the parameter space. If the loss is flat but the parameters keep moving widely, you're in a false plateau—the surface is merely noisy, not solved.

Two early signs: (1) the loss has zero variance across epoch boundaries, and (2) your learning rate has decayed to near zero while gradients remain large. That combo means you hit a structural barrier, not a minimum. Rewind, re-init with a wider distribution, and restart with a cosine schedule. Don't waste another epoch pretending the plateau is real.

Mini-FAQ: Quick Answers to Common Confusions

Does Adam make initialization irrelevant?

Short answer: no. Long answer: Adam adapts step sizes per parameter, which means it can survive a bad start — but "survive" is not the same as "thrive." I have seen teams throw Adam at a network initialized with standard deviations three times too large. The loss still dropped, eventually. But training took 40% longer, and the final validation accuracy sat two points lower than a run with sensible initialization and plain SGD. Adam masks the symptom, not the disease. It hides the chaotic surface behind a per-parameter learning rate. The catch is — once you need to debug convergence speed, or scale to a deeper model, that mask peels off. You end up chasing vanishing gradients that a proper init would have prevented in one line.

Adam buys you tolerance, not correctness. A bad init still leaves your optimizer climbing a steeper hill than necessary.

— A respiratory therapist, critical care unit

— internal comment from a production training pipeline, 2024

So tune the init first if you care about training speed or reproducibility across random seeds. Save Adam for the spiky losses where gradient variance dominates. Wrong order? You lose a day.

Should I tune LR first for transformers?

Transformers break the usual rules. Their loss surfaces are famously slippery — attention logits can explode from one batch to the next. Most teams I watch jump straight to learning rate tuning. That feels natural because the loss spikes visibly when LR is too high. But here is the trap: a transformer with a bad init produces those spikes even at moderate learning rates. We fixed this by scaling initialization according to the model width — not the default heuristic. Post-init, the same LR that previously caused loss spikes now ran smoothly. The learning rate was never the root cause; it was the magnifier. Tune LR first only if you have already verified that activations don't blow up or vanish at initialization. Otherwise you're adjusting the volume knob on a broken amplifier.

The pragmatic order for transformers? Validate init with a few forward passes — check mean activation magnitudes. Then tune LR on a tiny subset. That sequence saved us three restarts on a 1B-parameter model.

What about learning rate warmup?

Warmup is a band-aid — sometimes necessary, sometimes overkill. It compensates for the chaotic initial steps when gradient estimates are noisy and the optimizer is "blind." In a well-initialized network, warmup often does nothing. In a poorly initialized one, it hides the damage. I once ran an ablation where we kept a bad init but added a 10% warmup schedule. The loss curve looked fine for the first 500 steps. Then warmup ended, the full learning rate hit, and the loss jumped three orders of magnitude. The warmup had simply deferred the explosion. The real fix was correcting the initialization variance — after that we dropped warmup entirely and saw identical convergence. That hurts when you realize how many hours teams spend tuning warmup length.

Keep warmup as insurance for very deep models or huge batch sizes. But don't let it distract you from the real question: is the initial surface walkable without training wheels? Test it. If the loss drops cleanly at constant LR over the first 50 steps, skip the warmup. If it sputters, check init first — not warmup schedule.

Final Take: No Hype, Just the Messy Truth

When init wins

Start with initialization if your loss surface looks like a field of broken glass — sharp, disconnected gradients that spike or vanish within the first ten steps. I have seen teams burn weeks tuning learning rates on models that were simply born dead. A bad init doesn't just slow convergence; it locks you into a basin where no LR schedule can climb out. The fix is coarse: Xavier, He, or even a tiny orthogonal matrix. You're not hunting optimality yet — you're hunting *survival*. If your loss flatlines at a high value from epoch zero, reinit. That's the one case where I would bet on init before touching the LR. The catch is that init fixes are cheap but rare — most surfaces are not that broken.

When LR wins

Learning rate owns the fight when your loss curves look jagged but directional — bouncing around the same valley, never settling. That oscillating sawtooth is the signature of a step size that overshoots the minimum. We fixed this once by cutting the LR from 3e-4 to 1e-4, nothing else changed, and validation loss dropped ten points in an hour. Init was fine — the model *could* converge — it just couldn't land. The hard truth: LR tuning is endless. You will cycle through warmups, cosine decays, and one-cycle policies before admitting that 90% of the gain comes from picking the right magnitude, not the schedule. That sounds fine until you realize you have twenty runs and no clear winner.

But here is where the messy truth arrives — the two are not independent. Change init, and the optimal LR shifts. Jack the LR, and a once-stable init turns chaotic. Most of the time the blame goes to the wrong knob.

“We spent three weeks blaming the optimizer before noticing that a 10x smaller init made the same LR work flawlessly.”

— anonymous engineer from a production vision team I consulted with

One actionable rule of thumb

Run two short probes. First, freeze LR at 1e-3 and test three inits (default, smaller by factor 10, orthogonal). If any variant trains past a reasonable loss floor — congrats, init is not the bottleneck. Then keep the winning init and sweep LR over three orders of magnitude. The one that yields the steepest first-decade descent is your starter. Wrong order. Test LR first, and you might declare the architecture hopeless when it was just a bad init. That hurts. Not because the method is complex — it's not — but because the temptation to tweak the LR is always stronger. It feels like progress. Reinit feels like starting over. Sometimes starting over is exactly what you need.