Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Visual Entity Linking: A Preliminary Study

Weegar, Rebecka ; Hammarlund, Linus ; Tegen, Agnes ; Oskarsson, Magnus LU orcid ; Åström, Karl LU orcid and Nugues, Pierre LU orcid (2014) AAAI 2014 Workshop on Cognitive Computing for Augmented Human Intelligence p.46-49
Abstract
In this paper, we describe a system that jointly extracts

entities appearing in images and mentioned in their ac-

companying captions. As input, the entity linking pro-

gram takes a segmented image together with its cap-

tion. It consists of a sequence of processing steps: part-

of-speech tagging, dependency parsing, and coreference

resolution that enables us to identify the entities as well

as possible textual relations from the captions. The pro-

gram uses the image regions labelled with a set of pre-

defined categories and computes WordNet similarities

between these labels and the entity names. Finally, the

program links the entities it... (More)
In this paper, we describe a system that jointly extracts

entities appearing in images and mentioned in their ac-

companying captions. As input, the entity linking pro-

gram takes a segmented image together with its cap-

tion. It consists of a sequence of processing steps: part-

of-speech tagging, dependency parsing, and coreference

resolution that enables us to identify the entities as well

as possible textual relations from the captions. The pro-

gram uses the image regions labelled with a set of pre-

defined categories and computes WordNet similarities

between these labels and the entity names. Finally, the

program links the entities it detected across the text and

the images. We applied our system on the Segmented

and Annotated IAPR TC-12 dataset that we enriched

with entity annotations and we obtained a correct as-

signment rate of 55.48% (Less)
Please use this url to cite or link to this publication:
author
; ; ; ; and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
host publication
Cognitive computing for augmented human intelligence, papers presented at the Twenty-Eighth AAAI Conference on Artificial Intelligence
pages
4 pages
publisher
AAAI
conference name
AAAI 2014 Workshop on Cognitive Computing for Augmented Human Intelligence
conference dates
2014-07-27
external identifiers
  • scopus:84974688497
ISBN
978-1-57735-664-6
language
English
LU publication?
yes
id
07cb3525-9704-46c8-8e2c-deb259fcb965 (old id 4580516)
date added to LUP
2016-04-04 13:33:12
date last changed
2021-10-06 03:56:35
@inproceedings{07cb3525-9704-46c8-8e2c-deb259fcb965,
  abstract     = {In this paper, we describe a system that jointly extracts<br/><br>
entities appearing in images and mentioned in their ac-<br/><br>
companying captions. As input, the entity linking pro-<br/><br>
gram takes a segmented image together with its cap-<br/><br>
tion. It consists of a sequence of processing steps: part-<br/><br>
of-speech tagging, dependency parsing, and coreference<br/><br>
resolution that enables us to identify the entities as well<br/><br>
as possible textual relations from the captions. The pro-<br/><br>
gram uses the image regions labelled with a set of pre-<br/><br>
defined categories and computes WordNet similarities<br/><br>
between these labels and the entity names. Finally, the<br/><br>
program links the entities it detected across the text and<br/><br>
the images. We applied our system on the Segmented<br/><br>
and Annotated IAPR TC-12 dataset that we enriched<br/><br>
with entity annotations and we obtained a correct as-<br/><br>
signment rate of 55.48%},
  author       = {Weegar, Rebecka and Hammarlund, Linus and Tegen, Agnes and Oskarsson, Magnus and Åström, Karl and Nugues, Pierre},
  booktitle    = {Cognitive computing for augmented human intelligence, papers presented at the Twenty-Eighth AAAI Conference on Artificial Intelligence},
  isbn         = {978-1-57735-664-6},
  language     = {eng},
  pages        = {46--49},
  publisher    = {AAAI},
  title        = {Visual Entity Linking: A Preliminary Study},
  url          = {https://lup.lub.lu.se/search/files/19769215/4580517.pdf},
  year         = {2014},
}