FIM: Filling in the Middle for Language Models
Failed to add items
Add to cart failed.
Add to wishlist failed.
Remove from wishlist failed.
Follow podcast failed
Unfollow podcast failed
FIM: Filling in the Middle for Language Models
-
Narrated by:
-
Written by:
About this listen
This 2022 academic paper explores Fill-in-the-Middle (FIM) capabilities in causal decoder-based language models, demonstrating that these models can learn to infill text effectively by simply rearranging parts of the training data. The authors propose a method where a middle section of text is moved to the end of a document during training, showing this data augmentation does not negatively impact the model's original left-to-right generative ability. The research highlights the efficiency of FIM training, suggesting it should be a default practice, and offers best practices and hyperparameters for optimal performance, particularly noting the superiority of character-level span selection and context-level FIM implementation. They also introduce new benchmarks to evaluate infilling performance, emphasizing the importance of sampling-based evaluations over traditional perplexity measures for gauging real-world utility.
Source: https://arxiv.org/pdf/2207.14255