论文标题
生成想象力提升机器翻译
Generative Imagination Elevates Machine Translation
论文作者
论文摘要
在文本和图像上共享了常见的语义。用源语言给定句子，描绘视觉场景是否有助于翻译成目标语言？现有的多模式神经机器翻译方法（MNMT）需要双语句子的三联 - 训练图像和源句子的元素 - 推理图像。在本文中，我们提出了Imagit，这是一种通过视觉想象的新型机器翻译方法。 Imagit首先学会从源句子中生成视觉表示，然后同时利用源句子和“想象的表示”来产生目标翻译。与以前的方法不同，它只需要在推理时间的源句子。实验表明，图像从视觉想象中受益，并显着优于仅文本神经机器翻译基线。进一步的分析表明，在执行降解策略时，Imagit中的想象过程有助于填补丢失的信息。
There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the "imagined representation" to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time. Experiments demonstrate that ImagiT benefits from visual imagination and significantly outperforms the text-only neural machine translation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy.
