Linking Entities Across Images and Text
Weegar, Rebecka ; Åström, Karl LU
and Nugues, Pierre
LU
(2015)
Nineteenth Conference on Computational Natural Language Learning (CoNLL 2015)
p.185-193
- Abstract
- This paper describes a set of methods to link entities across images and text. As a corpus, we used a data set of images,
where each image is commented by a short caption and where the regions in the images are manually segmented and labeled with a category. We extracted the entity mentions from the captions and we computed a semantic similarity between the mentions and the region labels. We also
measured the statistical associations between these mentions and the labels and we combined them with the semantic similarity to produce mappings in the form of pairs consisting of a region label and
a caption entity. In a second step, we used the syntactic relationships between the mentions and the spatial... (More) - This paper describes a set of methods to link entities across images and text. As a corpus, we used a data set of images,
where each image is commented by a short caption and where the regions in the images are manually segmented and labeled with a category. We extracted the entity mentions from the captions and we computed a semantic similarity between the mentions and the region labels. We also
measured the statistical associations between these mentions and the labels and we combined them with the semantic similarity to produce mappings in the form of pairs consisting of a region label and
a caption entity. In a second step, we used the syntactic relationships between the mentions and the spatial relationships
between the regions to rerank the lists of candidate mappings. To evaluate our methods, we annotated a test set of 200 images, where we manually linked the im- age regions to their corresponding mentions in the captions. Eventually, we could match objects in pictures to their correct mentions for nearly 89 percent of the segments, when such a matching exists. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/7864937
- author
- Weegar, Rebecka
; Åström, Karl
LU
and Nugues, Pierre
LU
- organization
- publishing date
- 2015
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- host publication
- Proceedings of the Nineteenth Conference on Computational Natural Language Learning (CoNLL 2015)
- pages
- 185 - 193
- publisher
- Association for Computational Linguistics
- conference name
- Nineteenth Conference on Computational Natural Language Learning (CoNLL 2015)
- conference location
- Bejing, China
- conference dates
- 2015-07-30 - 2015-07-31
- external identifiers
-
- scopus:85072785215
- ISBN
- 978-1-941643-77-8
- language
- English
- LU publication?
- yes
- id
- eafeeff2-deb2-4739-96a1-b0b6927929f4 (old id 7864937)
- alternative location
- http://www.aclweb.org/anthology/K/K15/K15-1019.pdf
- date added to LUP
- 2016-04-04 14:09:09
- date last changed
- 2022-01-30 17:05:54
@inproceedings{eafeeff2-deb2-4739-96a1-b0b6927929f4,
abstract = {{This paper describes a set of methods to link entities across images and text. As a corpus, we used a data set of images,<br/><br>
where each image is commented by a short caption and where the regions in the images are manually segmented and labeled with a category. We extracted the entity mentions from the captions and we computed a semantic similarity between the mentions and the region labels. We also<br/><br>
measured the statistical associations between these mentions and the labels and we combined them with the semantic similarity to produce mappings in the form of pairs consisting of a region label and<br/><br>
a caption entity. In a second step, we used the syntactic relationships between the mentions and the spatial relationships<br/><br>
between the regions to rerank the lists of candidate mappings. To evaluate our methods, we annotated a test set of 200 images, where we manually linked the im- age regions to their corresponding mentions in the captions. Eventually, we could match objects in pictures to their correct mentions for nearly 89 percent of the segments, when such a matching exists.}},
author = {{Weegar, Rebecka and Åström, Karl and Nugues, Pierre}},
booktitle = {{Proceedings of the Nineteenth Conference on Computational Natural Language Learning (CoNLL 2015)}},
isbn = {{978-1-941643-77-8}},
language = {{eng}},
pages = {{185--193}},
publisher = {{Association for Computational Linguistics}},
title = {{Linking Entities Across Images and Text}},
url = {{https://lup.lub.lu.se/search/files/19770235/8626415.pdf}},
year = {{2015}},
}