Advancing
AI Safety Research

Truthful AI is a non-profit organization based in Berkeley, California and led by Owain Evans.
We do research towards the development of safe and aligned AI systems. Current topics include situational awareness, deception, and hidden reasoning in language models.

See Our Papers

Research Highlights

Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMS

A model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively.

Subliminal Learning:
Models Transmit Behavioral Traits Via Hidden Signals In Data

LLMs transmit traits to other models via hidden signals in data. Datasets consisting only of 3-digit numbers can transmit a love for owls, or evil tendencies.

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

If an LLM is trained on "Olaf Scholz was 9th Chancellor of Germany", it will not automatically be able to answer the question, "Who was 9th Chancellor of Germany?

Advancing AI Safety Research

Research Highlights

Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMS

Subliminal Learning: Models Transmit Behavioral Traits Via Hidden Signals In Data

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

Advancing
AI Safety Research

Subliminal Learning:
Models Transmit Behavioral Traits Via Hidden Signals In Data