Native Ternary Conversion of Pretrained Language Models via Optimal Transport Projection and Frozen-Code Distillation

1. Introduction

1.1 The Memory Bandwidth Wall & Ternary Weights

Autoregressive transformer inference is memory-bandwidth bound during token generation. For standard floating-point models (FP16/BF16), every token requires reading billions of parameters from VRAM across high-power memory buses:

Ternary representation $\{-1, 0, +1\}$ (the 1.58-bit regime pioneered by BitNet b1.58 [2], [3]) replaces floating-point multiply-accumulate (MAC) operations with integer additions and subtractions:

where $S_i^+ = \{j : W_{ij} = +1\}$ and $S_i^- = \{j : W_{ij} = -1\}$. This eliminates hardware multipliers, slashes memory footprint by $75\text{--}80\%$, and enables integer SIMD lookup kernels that run faster on standard CPUs than float matrix multiplications on GPUs.

1.2 The Conversion Bottleneck

While training 1.58-bit models from scratch is well-established, converting existing high-quality open-weight models (e.g., Qwen 2.5, Llama 3) to native ternary weights has remained an unsolved open challenge. Prior attempts encounter three fatal failure modes:

• Dead-Zone Gradient Starvation (STE Failure): In standard Straight-Through Estimation, pretrained float weights concentrated in the quantization dead zone $[-0.5 \Delta, +0.5 \Delta]$ produce exact zero ternary codes. Gradients backpropagated through STE fail to push parameters across quantization thresholds, leaving models stuck in float optimization basins (achieving only 1.7% ternary sparsity).
• Loss Landscape Instability (DLT Failure): Dual Learnable Ternarization (DLT) introduces learnable scaling and shifting thresholds, but backpropagation through continuous approximations on pretrained checkpoints causes logit KL divergence to overpower language modeling loss, destroying model syntax within 1,000 steps.
• Accumulated Activation Drift (OT-Only Failure): Direct mathematical projection onto ternary codes (without post-projection training) achieves low per-weight L1 error ($\approx 0.01$), but small errors compound across 28–80 transformer layers, producing a $25\%$ output activation shift and complete token degradation.

2. Methodology: Optimal Transport & Frozen-Code Distillation

2.1 1D Optimal Transport Projection

For each weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, we formulate ternary quantization as an Optimal Transport problem from the continuous empirical distribution of row $W_i$ to a discrete 3-point measure $\nu = w_{-1} \delta_{-s_i + \mu_i} + w_0 \delta_{\mu_i} + w_{+1} \delta_{+s_i + \mu_i}$:

In 1D, the optimal transport map is strictly monotonic and can be solved exactly via 1D Lloyd's k-means clustering per row:

1. Initialize: $\mu_i = \text{mean}(W_i)$, $s_i = \text{std}(W_i)$.
2. Assignment Step: Assign each weight $W_{ij}$ to the nearest centroid $c \in \{\mu_i - s_i, \, \mu_i, \, \mu_i + s_i\}$: $$C_{ij} = \arg\min_{k \in \{-1, 0, +1\}} \left| W_{ij} - (k \cdot s_i + \mu_i) \right|$$
3. Update Step: Recompute scale $s_i$ and shift $\mu_i$ via linear regression over assigned clusters: $$s_i = \frac{\sum_{j} C_{ij} (W_{ij} - \mu_i)}{\sum_j C_{ij}^2}, \quad \mu_i = \frac{1}{d_{\text{in}}} \sum_{j=1}^{d_{\text{in}}} (W_{ij} - s_i C_{ij})$$
4. Convergence: The algorithm converges in 10–20 iterations, producing an average per-weight L1 reconstruction error of $\approx 0.0102$.

2.2 GPU-Batched Vectorization

Previous CPU-bound implementations processed output rows sequentially via Python loops, requiring $\approx 45\text{ minutes}$ for a 7B model. We vectorize the entire Lloyd iteration across all rows simultaneously as tensor operations in PyTorch:

2.3 Frozen-Code Adaptation (TernaryFreezeLinear)

To prevent dead-zone gradient starvation, the quantized integer codes $C \in \{-1, 0, +1\}^{d_{\text{out}} \times d_{\text{in}}}$ are stored as immutable int8 tensor buffers (register_buffer). The optimizer is structurally prohibited from modifying the discrete codes:

In addition to channel-wise scales and shifts, we unfreeze:

• embed_tokens (input token embeddings)
• lm_head (output vocabulary projection)
• All LayerNorm / RMSNorm parameters (inter-layer activation calibration)

For Qwen 2.5 1.5B, this reduces trainable parameters from $1.54\text{B}$ down to $235\text{M}$ ($84.7\%$ parameter reduction during training).

2.4 Instruct Curriculum & Distillation

We train the continuous parameters across a structured 3-round curriculum with feature distillation:

3. Empirical Results & Analysis

3.1 Coherence Progression & Factual Recovery

3.2 The "Dip-Surge" Convergence Dynamic

Across both 1.5B and 7B curriculum fine-tuning runs, we observe a characteristic dip-surge dynamic:

3.3 Comparison with Alternative Conversion Paradigms

4. Inference Engine & Hardware Deployment

4.1 The i2_s_shifted GGUF Tensor Format

Standard ternary inference engines assume zero-centered weights without channel-wise shifts. Dropping the shift degrades model coherence from $8/8$ down to $2/8$. We define the extended i2_s_shifted binary layout:

where $\sum_j x_j$ is computed once per forward pass. The row shift adds only one scalar multiply-add per output row — introducing negligible ($< 0.1\%$) computational overhead while preserving $100\%$ factual accuracy.

4.2 CPU/GPU Split Architecture for Consumer Workstations

Ternary models reverse traditional hardware balance: integer SIMD additions run faster on modern CPUs (AVX2 / AVX-512) than floating-point matrix multiplications, while GPUs excel at memory-heavy float attention and KV-cache maintenance.

5. Ternary Patch Adaptation (TPAT) & In-Place Bit-Flips

Because converted models store weights as discrete ternary values $\{-1, 0, +1\}$, subsequent parameter-efficient adaptation can be executed directly as binary bit-flips ($\Delta W_{ij} \in \{-2, -1, 0, +1, +2\}$). Using memory-mapped file descriptors (mmap), specialized domain patches (TPATs) modify $100,000$ weight coordinates directly in disk-backed VRAM in under $25\text{ ms}$ without model reloading.

Furthermore, ternary networks exhibit an extreme amplification tolerance ceiling ($\gamma > 500$, compared to $\gamma \approx 35\text{--}42$ in float models), enabling aggressive post-training domain reinforcement without syntactic degeneration.

6. Conclusion & Roadmap

We have demonstrated that dense, pretrained Large Language Models can be converted to native 1.58-bit ternary representations without retraining from scratch. By combining GPU-batched 1D Optimal Transport projection with frozen-code instruct distillation, models preserve full factual integrity while achieving $4.5\times$ memory compression and zero-multiplication integer arithmetic. Future work will scale this pipeline to 31B+ architectures (Gemma 2 27B, Llama 3 70B) and integrate TurboQuant 3-bit KV-cache compression for fully local, sub-4GB mobile deployments.

References

1. [1] Hu, E. J., et al. (2022). "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022.
2. [2] Wang, H., et al. (2024). "BitNet: Scaling 1-bit Transformers for Large Language Models." arXiv:2310.11453.
3. [3] Ma, S., et al. (2024). "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits." arXiv:2402.17764.
4. [4] Martin, J. (2026). "Architecture-Compatible Sparse Knowledge Transfer for Large Language Models." Cross Domain Reasoning Technical Report CDR-TR-2026-02.
5. [5] Peyré, G., & Cuturi, M. (2019). "Computational Optimal Transport." Foundations and Trends in Machine Learning.

Citation

@techreport{martin2026ternaryconversion,
  title={Native Ternary Conversion of Pretrained Language Models via Optimal Transport Projection and Frozen-Code Distillation},
  author={Martin, J.},
  institution={Cross Domain Reasoning},
  year={2026},
  month={August},
  number={CDR-TR-2026-01},
  doi={10.5281/cdr.2026.01},
  url={https://crossdomainreasoning.com/papers/ternary-conversion/}
}