neuralese

Definition of neuralese

1. : the internal numerical 'language' a model might use to think and communicate, which humans cannot read; often discussed as a risk if future models stop reasoning in plain words

   • If models start reasoning in neuralese, we lose the ability to read their chain of thought.
   • Safety researchers want models to keep thinking in English, not neuralese.