OLIVAW is an AI Othello playing agent which autonomously learns how to improve its gameplay by playing against itself. Some top-notch players (including former World Champions) reported that they had the impression that OLIVAW's gameplay was human-like. To better investigate the processes related to these impressions, we conducted a pilot study using the Othello Game Evaluation App, a computer application we developed to evaluate pre-recorded Othello games in a controlled setting while assuring an adequate user experience. An exploratory analysis of the results shows that the participants mostly evaluated OLIVAW as a human. When asked for a motivation for their choice, some of them reported that they evaluate poor game moves (and, consequently, losing the game) as an indication of the human-likeness of the player.
