Counting Letters, Chaining Premises & Solving Equations: Exploring Inverse Scaling Problems with GPT-3

Language models generally show increased performance in a variety of tasks as their size increases. But there are a class of problems for which increase in model size results in worse performance. These are known as inverse scaling problems. In this work, we examine how GPT-3 performs on tasks that involve the use of multiple, interconnected premises and those that require the counting of letters within given strings of text as well as solving simple multi-operator mathematical equations.