Md Rifat Arefin

About

I’m a final-year Ph.D. candidate at Mila and Université de Montréal, advised by Irina Rish, and have collaborated with Yoshua Bengio and Yann LeCun. I study why large neural networks learn the representations they do, and how architecture and optimization shape them. My work combines theory, mostly tensor decomposition and random matrix theory, with controlled experiments on real language models.

Current threads:

• Optimization dynamics in pretraining: how temporary interventions keep changing training long after they end (preprint, 2026).
• Expressivity of transformer MLPs: what gated blocks like SwiGLU can represent that tied blocks like ReLU² cannot (CP-MLPs, NeurIPS 2026).
• Memory in sequence models: delta-rule and linear-attention architectures, and KV-cache compression (TT-Delta, COLM 2026 workshop).
• Representation geometry: collapse and compression across layers, and their link to reasoning (Seq-VCR, ICLR 2025; Layer by Layer, ICML 2025).

News

• Oct 2026New preprint from my Meta internship: Delayed Effects of Temporary Weight Decay in Muon Pretraining.
• Sep 2026When Redundant Probes Help in Strategic Auditing accepted at the EvoRobust Workshop @ NeurIPS 2026.
• Sep 2026CP-MLPs, a tensor-rank theory of tied vs. untied MLP blocks, accepted at NeurIPS 2026.
• Sep 2026TT-Delta (KV-cache compression) accepted at the Efficient Reasoning Workshop @ COLM 2026.
• Jun 2026Started a second research internship at Meta.
• Jan 2026Finished my research internship at Meta.

Selected publications

• 
  Delayed Effects of Temporary Weight Decay in Muon Pretraining

  Md Rifat Arefin, Ravid Shwartz-Ziv, Ernie Chang, Chinnadhurai Sankar, Rylan Conway, Aristide Baratin, Irina Rish, Adithya Sagar, Patrick Huber

  Preprint, 2026 Work done at Meta

  A weight-decay window that is 0.26 nats worse when it ends finishes 0.028 nats better at 1.4B, because the weight-norm deficit it leaves keeps relative steps larger long after removal.

  PDF
• 
  CP-MLPs: A Tensor-Rank Theory of Tied and Untied MLP Blocks

  Md Rifat Arefin, Farzaneh Heidari, Irina Rish, Guillaume Rabusseau

  Neural Information Processing Systems (NeurIPS), 2026

  Untying the detector and value directions (SwiGLU vs. ReLU²) gives exact width separations: untied units need fewer channels for indefinite and role-asymmetric interactions; tied units match pure powers.

  PDF
• COLM-W
  TT-Delta: Dual-Channel KV-Cache Compression with Tensor-Train and Delta-Rule State

  Md Rifat Arefin

  Efficient Reasoning Workshop @ COLM, 2026

  A KV-cache codec that pairs a tensor-train structural channel with a delta-rule associative channel.

  OpenReviewPDF
• 
  Layer by Layer: Uncovering Hidden Representations in Language Models

  Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, Ravid Shwartz-Ziv

  International Conference on Machine Learning (ICML), 2025 Spotlight

  Intermediate layers often give better features than the final layer; we explain when and why using information-theoretic and geometric metrics.

  arXivPDF
• 
  Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning

  Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, Christopher Pal

  International Conference on Learning Representations (ICLR), 2025

  Representation collapse in intermediate layers limits multi-step arithmetic reasoning; a variance-covariance regularizer prevents it.

  arXivPDFCode
• 
  Unsupervised Concept Discovery Mitigates Spurious Correlations

  Md Rifat Arefin, Yan Zhang, Aristide Baratin, Francesco Locatello, Irina Rish, Dianbo Liu, Kenji Kawaguchi

  International Conference on Machine Learning (ICML), 2024

  Object-centric concept discovery groups training data without group labels, improving worst-group accuracy under spurious correlations.

  arXivCode

Industry experience

• Meta, Research Scientist Intern (second internship) Jun 2026 – Nov 2026
• Meta, Research Scientist Intern Aug 2025 – Jan 2026
• Samsung AI Lab, Montreal, Research Intern Feb 2025 – Jul 2025
• Amazon, Seattle, Applied Scientist Intern Oct 2024 – Jan 2025
• ServiceNow Research, Visiting Researcher May 2024 – Sep 2024
• Recursion Pharmaceuticals, Research Intern Jun 2023 – Dec 2023

Service and awards

Reviewer: NeurIPS 2024, 2025, 2026; ICML 2025, 2026; ICLR 2025, 2026, 2027; AISTATS 2024

• Notable Reviewer, ICLR 2025
• IVADO Ph.D. Excellence Scholarship (2021–2024)
• 1st runner-up, PROBA-V Super-Resolution Challenge, European Space Agency (team “rarefin”)
• Travel grant, Deep Learning and Reinforcement Learning Summer School 2019
• President's Scholarship, Master's studies
• IEEE BDS Best Paper Award, International Conference on Computer and Information Technology (ICCIT) 2015