When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP

A new academic study finds that large language models frequently break working legacy code when asked to modernize it, often performing worse than leaving the code untouched. Researchers tested five LLMs on 100 WordPress 4.0 PHP files, asking each to upgrade the code to PHP 8.3 without project context or execution feedback, then evaluated outputs against 630 hidden behavioral checks and 511 Rector-detected modernization findings. The results showed that while 83 of the original files already passed all checks, the best-performing LLM left only 59 intact, with each model breaking 32-51 working files while repairing at most 10; by comparison, the rule-based tool Rector broke 7 files and repaired 6. The damage was largely avoidable, as only 37 findings (7 percent) actually required changes under PHP 8.3, yet LLMs broke 36-60 percent of already-compatible files (versus 8 percent for Rector), with most failures stemming from files that no longer loaded due to invalid type declarations. The study's authors, including lead submitter Afridi Mahmud, argue that LLM-based modernization evaluations should measure against a no-change baseline and report modernization and behavioral preservation separately. The findings arrive as enterprises weigh AI coding assistants against rule-based tools for legacy migration projects, with attention likely to turn to whether results hold with project context and execution feedback included.