Post by PRATIKKUMAR PRAVINBHAI BABARIYA

I tried to reproduce my own result. It didn't work. The asymmetry I found — AttnOnly converging while FFNOnly collapsed — held perfectly across three seeds in the original environment. Val loss 1.600 ± 0.015. Tight. Consistent. I trusted it. Then I moved to new hardware, forced deterministic CUDA execution, and reran everything. FFNOnly reproduced to four decimal places. Every seed. Every machine. The collapse is real. AttnOnly gave 2.591 ± 0.544. One seed hit 3.354 — the FFNOnly collapse floor itself. At 124M, two additional seeds approached the floor from below. Same code. Same seeds. Different environment. Different result. I spent weeks trying to explain the gap. Ruled out SDPA backend differences. Ruled out run-to-run non-determinism. The original software environment wasn't preserved, so the exact source remains unresolved. Here's what I know for certain: FFNOnly collapse is robust. Here's what I don't: whether AttnOnly recovery is a real architectural property or an environment-sensitive artifact I happened to observe once. #MachineLearning #Transformers #Research #DeepLearning