Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer

Huiyuan Lai; Antonio Toral Ruiz; Malvina Nissim

doi:10.18653/v1/2021.acl-short.62

Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer

Huiyuan Lai, Antonio Toral Ruiz, Malvina Nissim

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

35 Citations (Scopus)

164 Downloads (Pure)

Abstract

Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content –the two core aspects of the task– we achieve a new state-of-the-art.

Original language	English
Title of host publication	Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing
Editors	Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Place of Publication	Bangkok, Thailand
Publisher	Association for Computational Linguistics, ACL Anthology
Pages	484-494
Number of pages	11
Volume	2
DOIs	https://doi.org/10.18653/v1/2021.acl-short.62
Publication status	Published - 2021

Access to Document

10.18653/v1/2021.acl-short.62Licence: CC BY

Thank you BART! Rewarding Pre-Trained Models Improves Formality Style TransferFinal publisher's version, 453 KBLicence: CC BY

Handle.net

Persistent link

Cite this

Lai, H., Toral Ruiz, A., & Nissim, M. (2021). Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer. In C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol. 2, pp. 484-494). Association for Computational Linguistics, ACL Anthology. https://doi.org/10.18653/v1/2021.acl-short.62

Lai, Huiyuan ; Toral Ruiz, Antonio ; Nissim, Malvina. / Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. editor / Chengqing Zong ; Fei Xia ; Wenjie Li ; Roberto Navigli. Vol. 2 Bangkok, Thailand : Association for Computational Linguistics, ACL Anthology, 2021. pp. 484-494

@inproceedings{81df322b1fb6475d9036a75460301a7c,

title = "Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer",

abstract = "Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content –the two core aspects of the task– we achieve a new state-of-the-art.",

author = "Huiyuan Lai and \{Toral Ruiz\}, Antonio and Malvina Nissim",

year = "2021",

doi = "10.18653/v1/2021.acl-short.62",

language = "English",

volume = "2",

pages = "484--494",

editor = "Chengqing Zong and Fei Xia and Wenjie Li and Roberto Navigli",

booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing",

publisher = "Association for Computational Linguistics, ACL Anthology",

}

Lai, H, Toral Ruiz, A & Nissim, M 2021, Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer. in C Zong, F Xia, W Li & R Navigli (eds), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. vol. 2, Association for Computational Linguistics, ACL Anthology, Bangkok, Thailand, pp. 484-494. https://doi.org/10.18653/v1/2021.acl-short.62

Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer. / Lai, Huiyuan; Toral Ruiz, Antonio; Nissim, Malvina.
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ed. / Chengqing Zong; Fei Xia; Wenjie Li; Roberto Navigli. Vol. 2 Bangkok, Thailand: Association for Computational Linguistics, ACL Anthology, 2021. p. 484-494.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer

AU - Lai, Huiyuan

AU - Toral Ruiz, Antonio

AU - Nissim, Malvina

PY - 2021

Y1 - 2021

N2 - Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content –the two core aspects of the task– we achieve a new state-of-the-art.

AB - Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content –the two core aspects of the task– we achieve a new state-of-the-art.

U2 - 10.18653/v1/2021.acl-short.62

DO - 10.18653/v1/2021.acl-short.62

M3 - Conference contribution

VL - 2

SP - 484

EP - 494

BT - Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

A2 - Zong, Chengqing

A2 - Xia, Fei

A2 - Li, Wenjie

A2 - Navigli, Roberto

PB - Association for Computational Linguistics, ACL Anthology

CY - Bangkok, Thailand

ER -

Lai H, Toral Ruiz A, Nissim M. Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer. In Zong C, Xia F, Li W, Navigli R, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Vol. 2. Bangkok, Thailand: Association for Computational Linguistics, ACL Anthology. 2021. p. 484-494 doi: 10.18653/v1/2021.acl-short.62

Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer

Abstract

Access to Document

Handle.net

Fingerprint

Cite this