ProjectResearch prototype

DoRA × KAN: Nonlinear Weight Adapters

Linear adapters are not the only way to move a model.

Documented February 14, 2025

Why I made it

I explored this because I wanted to understand whether the directional update in DoRA could express more than a fixed linear map without abandoning parameter-efficient fine-tuning.

DoRA decomposes pretrained weights into magnitude and direction, then applies LoRA to the directional component. This project replaces the linear low-rank adapter with a KAN, using learnable B-spline activations instead of fixed linear projections. The result is a PEFT method that captures nonlinear relationships in the weight residual space, adapting patterns that vanilla LoRA systematically misses.

Tools: PyTorch · DoRA · KAN · LoRA · PEFT

Parameter-efficient fine-tuning (PEFT) methods exist because full fine-tuning a modern LLM is wasteful: most of the pretrained weight matrix is already correct, and you only need to nudge it. The question is what shape that nudge should take.

The problem

LoRA's answer is: a low-rank linear update, W' = W + BA, where B and A are small matrices. This works, but it bakes in two assumptions: that the useful update is low-rank and that it is linear.

DoRA (Weight-Decomposed Low-Rank Adaptation) relaxes the first assumption partially: it decomposes the pretrained weight into a magnitude scalar and a direction vector, then applies LoRA only to the direction. This better matches how full fine-tuning actually moves weights, but the directional update itself is still a linear, low-rank projection.

This project asks: what if the directional update is allowed to be nonlinear?Worth stating the honest prior: the linear assumption is not obviously wrong, and LoRA works extremely well in practice. This is a question about the edge cases, not a claim that the standard approach is broken.

How it works

The diagram above shows the full decomposition. Concretely:

Decompose the pretrained weight W into magnitude m = ||W|| and direction V = W / ||W||.
Adapt the direction with a KAN. Instead of a linear BA projection, the directional residual ΔV is produced by a Kolmogorov-Arnold Network layer, a small network whose activation functions are learnable B-splines on each edge, rather than fixed nonlinearities (ReLU/GELU) applied after fixed linear weights.
Recompose the adapted weight: W' = m · (V + ΔV) / ||V + ΔV||, renormalizing so the direction stays a unit vector and the magnitude stays under separate, explicit control.

The intuition: a linear adapter can only stretch, rotate, and shear the residual space it operates in. A B-spline-based adapter can bend it, so directions in weight-residual space that require a nonlinear correction (which vanilla LoRA/DoRA can only approximate by stacking more rank) can be captured with fewer parameters.

Adapter family comparisonWhat each method adapts, and what kind of function it can represent.

Method	Adapts	Update function class	Extra params vs. LoRA
LoRA	Full weight W directly	Linear, low-rank (BA)	Baseline
DoRA	Direction V (magnitude m frozen-ish, learned separately)	Linear, low-rank on V	+1 magnitude vector per layer
DoRA × KAN (this project)	Direction V via KAN layer	Nonlinear — learnable B-spline per edge	+ spline coefficients per edge (small, configurable grid size)

Key design decisions

KAN only on the directional component, not the whole weight. Applying a KAN to the full W would be far too expensive: W is the entire pretrained matrix. By restricting the nonlinearity to the much smaller directional residual, the parameter overhead stays in PEFT territory (low single-digit percent of base model size), while the expressivity of that residual goes up.
Renormalization after adaptation. Without renormalizing V + ΔV back to a unit vector, the magnitude/direction split degenerates: the KAN could just learn to scale the vector, which is exactly the behavior DoRA was designed to separate out and control explicitly.
B-splines over MLPs for ΔV. A small MLP could also be nonlinear, but B-spline activations are locally supported (each spline only responds to a region of its input range), which keeps the adapter's behavior interpretable per-input-region and avoids the global entanglement that makes small MLP adapters hard to initialize stably.

Status

This is an active research prototype. The architecture and the recomposition math above are implemented; the open work is systematic benchmarking against vanilla LoRA and DoRA across a range of adaptation tasks (instruction-following, domain adaptation, and arithmetic/reasoning probes, the kinds of tasks where "nonlinear relationships in the weight residual space" should matter most). I'm tracking training stability (the renormalization step is numerically delicate at low precision) as closely as task accuracy. No code is public yet; this section will link to a repo once there's something worth benchmarking against.

Limitations

Spline grid size is a new hyperparameter. KAN layers introduce a grid resolution for the B-splines: too coarse and you lose the nonlinearity benefit, too fine and you reintroduce the parameter bloat PEFT is meant to avoid.
No claim of universal improvement. For tasks where the optimal weight update genuinely is close to linear, DoRA × KAN should converge to behave like plain DoRA with extra unused capacity; the hypothesis is about which tasks benefit, not that this should replace LoRA/DoRA everywhere.

For background on where PEFT sits in the broader spectrum of teaching a model new behavior, see Fine-tuning is a last resort, not a first move.