Advanced

Word length, sentence length and frequency: Zipf's law revisited.

Sigurd, Bengt LU ; Eeg-Olofsson, Mats LU and van de Weijer, Joost LU (2004) In Studia Linguistica 58(1). p.37-52
Abstract
This paper examines data from English, Swedish and German in order to find a theoretical distribution that describes the observed relation between word length and frequency. In Swedish and English, most word tokens consist of three letters only, while shorter or longer words occur less frequently. We found that the equation with the general form fexp = a * Lb * cL (a variant of the so-called gamma distribution) approximates the observed frequencies reasonably well. This formula incorporates both the fact that the number of possible words increases with word length, and the fact that longer words tend to be avoided, presumably because they are uneconomic. To our knowledge this formula has not been proposed to describe word frequency data.... (More)
This paper examines data from English, Swedish and German in order to find a theoretical distribution that describes the observed relation between word length and frequency. In Swedish and English, most word tokens consist of three letters only, while shorter or longer words occur less frequently. We found that the equation with the general form fexp = a * Lb * cL (a variant of the so-called gamma distribution) approximates the observed frequencies reasonably well. This formula incorporates both the fact that the number of possible words increases with word length, and the fact that longer words tend to be avoided, presumably because they are uneconomic. To our knowledge this formula has not been proposed to describe word frequency data. We examined frequency distributions of word length in Swedish and English, and explored different variants of the equation by systematically varying the a, b and c parameters. Subsequently, we also applied the formula to the frequency distribution of sentence length in English, and found an almost perfect fit for a corpus consisting of different text genres. Moreover, the data showed that the formula can be used to distinguish between different kinds of text genres. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
length frequency
in
Studia Linguistica
volume
58
issue
1
pages
37 - 52
publisher
Wiley-Blackwell
external identifiers
  • wos:000220654000003
ISSN
1467-9582
DOI
10.1111/j.0039-3193.2004.00109.x
language
English
LU publication?
yes
id
c4d5330e-e441-4776-a7a7-de0418859186 (old id 161894)
date added to LUP
2007-07-27 14:58:58
date last changed
2016-04-16 03:02:29
@article{c4d5330e-e441-4776-a7a7-de0418859186,
  abstract     = {This paper examines data from English, Swedish and German in order to find a theoretical distribution that describes the observed relation between word length and frequency. In Swedish and English, most word tokens consist of three letters only, while shorter or longer words occur less frequently. We found that the equation with the general form fexp = a * Lb * cL (a variant of the so-called gamma distribution) approximates the observed frequencies reasonably well. This formula incorporates both the fact that the number of possible words increases with word length, and the fact that longer words tend to be avoided, presumably because they are uneconomic. To our knowledge this formula has not been proposed to describe word frequency data. We examined frequency distributions of word length in Swedish and English, and explored different variants of the equation by systematically varying the a, b and c parameters. Subsequently, we also applied the formula to the frequency distribution of sentence length in English, and found an almost perfect fit for a corpus consisting of different text genres. Moreover, the data showed that the formula can be used to distinguish between different kinds of text genres.},
  author       = {Sigurd, Bengt and Eeg-Olofsson, Mats and van de Weijer, Joost},
  issn         = {1467-9582},
  keyword      = {length frequency},
  language     = {eng},
  number       = {1},
  pages        = {37--52},
  publisher    = {Wiley-Blackwell},
  series       = {Studia Linguistica},
  title        = {Word length, sentence length and frequency: Zipf's law revisited.},
  url          = {http://dx.doi.org/10.1111/j.0039-3193.2004.00109.x},
  volume       = {58},
  year         = {2004},
}