Are We There Yet? Contextual Effects of Computer Model Judgments of Similarity vs. Human Judgments

Abstract

Similarity is used every day to help people organize the world and make classifications and generalizations. Quantitative models used to study similarity, including vector space models, represent words as vectors in multi-dimensional space and determine the similarity between two words by calculating the cosine of the angle between the two vectors. This thesis demonstrates that existing models do not reproduce human judgments of non-transitive aspects of similarity: specifically, the effect of order or context. I investigate if humans judge two words as more similar after a “context” pair of words or after a random pair of words, using Amazon’s Mechanical Turk crowd-sourcing platform. Human judgments are found to be asymmetric, with similarity increased when a relevant context is provided by the context pair. Word2Vec, a well-known vector space model that uses cosine distance to calculate similarity between pairs of words, is unaffected by other comparisons and therefore is unable to capture the effect of context. These results confirm that humans take context into account when judging between words, whereas Word2Vec does not. I suggest a neural circuit, which includes regions implicated in the semantic circuit, that may be involved in the similarity judgment task as performed by humans.