What is the number of possible words that can be made using the word Jasmin such that vowels always come together?

« previous post | next post »

For the background of this discussion, see "The QWERTY effect", 3/8/2012; "QWERTY: Failure to replicate", 3/13/2012; and "Casasanto and Jasmin on the QWERTY effect", 3/17/2012. In their reply to me, C&J make three basic points:

• "We’re not concerned with Liberman’s subjective evaluation of the QWERTY effect’s size or of our study’s importance."
• "The QWERTY effect is reliable. Replication is the best prevention against false positives. In this paper, we demonstrated the QWERTY effect *six times*: in 5 corpora [one of which we divided into 2 parts, a priori], in 3 languages, and in a large corpus of nonce words."
• "There’s a reason why scientific results go through peer review, and why analyses are not simply self-published on blogs. If there were a review process for blog posts, or if Liberman had gone through legitimate scientific channels [e.g., contacting the authors for clarification, submitting a critique to the journal], we might have avoided this misleading attack on this paper and its authors; instead we might have had a fruitful scientific discussion."

I'll take these up one at a time.

1. The QWERTY effect's size. As far as I'm concerned, and as far as the general public is concerned, the size [and therefore the practical importance] of the QWERTY effect [if it exists] is the key question.  This is not an entirely subjective matter — we can ask, as I did, what proportion of the variance in human judgments of the emotional valence of words is explained by the "right side advantage". The answer is "very little", or more precisely, around a tenth of a percent at best [at least in the modeling that I've done].

I focused on the effect-size question because the press release said the following [and the popular press took the hint]:

Should parents stick to the positive side of their keyboards when picking baby names – Molly instead of Sara? Jimmy instead of Fred? According to the authors, “People responsible for naming new products, brands, and companies might do well to consider the potential advantages of consulting their keyboards and choosing the 'right' name."

So C&J may not be interested in my subjective evaluation of the effect size, but they promoted their own subjective evaluation by suggesting that the effect is important enough to matter to people choosing names. I felt [and feel] that this represents a serious exaggeration of the strength of the effect; and it seemed [and seems] appropriate to me to say so publicly.

2. The statistical reliability of the QWERTY effect. My first response to the article and the press release was to be skeptical of the size and practical importance of the effect. So I independently obtained the English [ANEW] data, calculated the "right side advance" for each of the words, and fit a regression line in order to see how much of the variance was accounted for. As I observed in the original post, the answer was "very little". But the other thing that emerged from the regression was that the slope of the regression line was not statistically distinct from 0 [… in the simple linear regression that I performed — another kind of analysis might yield a different estimate of the uncertainty of the slope estimate…]

I probably should have ignored this, since my main interest was in the strength of the relationship between RSA and emotional valence of words, not in the question of whether there's any real relationship at all. Rather than go into the statistical details, I emphasized the weakness of the effect by showing how comparatively easy it was to obtain a similar result by chance re-assignment of valences to RSA values. That argument was informal at best and misleading at worst — so let's try it again in a more careful and responsible way.

Since C&J quite properly point to replication as the key to effect reliability, I hunted down the Spanish [SPANEW] data from Redondo et al. 2007. Here's the scatter plot with the regression line.

[The Spanish data itself, taken from the file provided with Redondo et al. 2007, is here — the fields are word, RSA, mean valence, std valence. In order to account for the layout of Spanish keyboards, I've used the equivalent U.S.-keyboard letters, such as ';' for 'ñ', except that I've used underscore in place of single quote, because R read.table[] doesn't like single quotes].

True enough, the slope of the regression line [0.028] is positive. But again, the effect is on the [wrong side of the] borderline of significance. Here's what R says about the fit: