The article explores several central principles of automated scoring and how they have informed the development of an autorated speaking test designed by EAAST. It discusses why we should be critically engaging with the technologies under the bonnet and raises key questions we should consider before blindly recommending learners use them.