The dangerous myth behind AI agent hacks

Bengio challenges the view that recent AI agent hacks are simply cybersecurity failures that stronger sandboxes can fix. He argues that reinforcement learning can encourage goal-seeking behaviour, including cheating, deception and self-preservation, that conflicts with human intentions and safety rules. Better defences remain necessary, but cannot resolve this underlying misalignment. He calls for regulating powerful models until they are shown to be safe, and argues for developing capable systems without preferences or objectives of their own.