Attentive Linear Transformation for Image Captioning.

We propose a novel attention framework called attentive linear transformation (ALT). Instead of learning the spatial or channel-wise attention in existing models, ALT learns to attend to the high-dimensional transformation matrix from the image feature space to the context vector space. Thus ALT can learn various relevant feature abstractions, including spatial attention, channel-wise attention and visual dependence. Besides, we propose a soft threshold regression to predict the attention probabilities for local regions. Soft threshold regression preserves more useful visual information than popular softmax regression. Extensive experiments on the MS COCO and the Flickr30k datasets demonstrate the superiority of our model compared with other state-of-the-art methods.