publications

2025

1. Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

   Mathieu Blondel , Michael E. Sander , Germain Vivier-Ardisson, Tianlin Liu , and 1 more author

   2025

   Bib

   @article{blondel2025autoregressivelanguagemodelssecretly,
     title = {Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction},
     author = {Blondel, Mathieu and Sander, Michael E. and Vivier-Ardisson, Germain and Liu, Tianlin and Roulet, Vincent},
     year = {2025},
     eprint = {2512.15605},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
     url = {https://arxiv.org/abs/2512.15605},
   }

2. 

   Learning with Local Search MCMC Layers

   Germain Vivier-Ardisson, Mathieu Blondel , and Axel Parmentier

   arXiv preprint arXiv:2505.14240, May 2025

   arXiv:2505.14240 [cs]

   Abs Bib

   Integrating combinatorial optimization layers into neural networks has recently attracted significant research interest. However, many existing approaches lack theoretical guarantees or fail to perform adequately when relying on inexact solvers. This is a critical limitation, as many operations research problems are NP-hard, often necessitating the use of neighborhood-based local search heuristics. These heuristics iteratively generate and evaluate candidate solutions based on an acceptance rule. In this paper, we introduce a theoretically-principled approach for learning with such inexact combinatorial solvers. Inspired by the connection between simulated annealing and Metropolis-Hastings, we propose to transform problem-specific neighborhood systems used in local search heuristics into proposal distributions, implementing MCMC on the combinatorial space of feasible solutions. This allows us to construct differentiable combinatorial layers and associated loss functions. Replacing an exact solver by a local search strongly reduces the computational burden of learning on many applications. We demonstrate our approach on a large-scale dynamic vehicle routing problem with time windows.

   @article{vivier2025learning,
     title = {Learning with Local Search MCMC Layers},
     author = {Vivier-Ardisson, Germain and Blondel, Mathieu and Parmentier, Axel},
     journal = {arXiv preprint arXiv:2505.14240},
     urldate = {2025-05-21},
     publisher = {arXiv},
     month = may,
     year = {2025},
     note = {arXiv:2505.14240 [cs]},
     keywords = {Computer Science - Machine Learning},
     url = {https://arxiv.org/abs/2505.14240}
   }

2024

1. 

   CF-OPT: Counterfactual Explanations for Structured Prediction

   Germain Vivier-Ardisson, Alexandre Forel , Axel Parmentier , and Thibaut Vidal

   May 2024

   arXiv:2405.18293 [cs]

   Abs Bib Code

   Optimization layers in deep neural networks have enjoyed a growing popularity in structured learning, improving the state of the art on a variety of applications. Yet, these pipelines lack interpretability since they are made of two opaque layers: a highly non-linear prediction model, such as a deep neural network, and an optimization layer, which is typically a complex black-box solver. Our goal is to improve the transparency of such methods by providing counterfactual explanations. We build upon variational autoencoders a principled way of obtaining counterfactuals: working in the latent space leads to a natural notion of plausibility of explanations. We finally introduce a variant of the classic loss for VAE training that improves their performance in our specific structured context. These provide the foundations of CF-OPT, a first-order optimization algorithm that can find counterfactual explanations for a broad class of structured learning architectures. Our numerical results show that both close and plausible explanations can be obtained for problems from the recent literature.

   @article{vivier--ardisson_cf-opt_2024,
     title = {{CF}-{OPT}: {Counterfactual} {Explanations} for {Structured} {Prediction}},
     shorttitle = {{CF}-{OPT}},
     url = {http://arxiv.org/abs/2405.18293},
     urldate = {2024-05-29},
     publisher = {arXiv},
     author = {Vivier-Ardisson, Germain and Forel, Alexandre and Parmentier, Axel and Vidal, Thibaut},
     month = may,
     year = {2024},
     note = {arXiv:2405.18293 [cs]},
     keywords = {Computer Science - Machine Learning},
   }