Choosing the best tools for comparative analyses of texts

What measurements should linguists use when comparing texts written by different writers? We report aspects of a systematic evaluation of 381 different language measures derived from 200 analytic tools, carried out during the pilot for a study exploring genetic contributions to language variation. The measures covered lexis, structure, meaning, and discourse features, and were evaluated with a focus on capturing numerically the qualitative features that linguists consider central to differentiating one text from another. We review principles for selecting analytic tools, and the choices faced by the researcher in processing and analysing data. We then identify and demonstrate five of the measures, which between them provide a useful profile of different linguistic features, and note correlations with psychometric measures taken for each writer. We conclude with some caveats regarding general issues of validity and some indications about potential links between our work and research into authorship attribution for forensic purposes.