Skip to content
EMA AI Standard v1.0

Sources

Every work the EMA AI Body of Knowledge cites. Each is real and checkable; where a finding is disputed, the unit citing it says so.

← EMA AI Standard
  • Daron Acemoglu (2025). The Simple Macroeconomics of AI. Economic Policy 40(121). Link
    a task-based estimate of AI's effect on total factor productivity over ten years, far below popular forecasts
  • John Allspaw (2012). Blameless PostMortems and a Just Culture. Etsy, Code as Craft (engineering blog).
    the case for blameless incident reviews; the distinction between blamelessness and the absence of accountability
  • Berk Atıl, Sarp Aykent, Alexa Chittams et al. (2025). Non-Determinism of "Deterministic" LLM System Settings in Hosted Environments. Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP). Link
    hosted models set to be deterministic varied in accuracy by up to 15% across identical runs
  • Alberto Bacchelli and Christian Bird (2013). Expectations, Outcomes, and Challenges of Modern Code Review. Proceedings of the 35th International Conference on Software Engineering (ICSE), 712–721.
    review's benefits are knowledge transfer and awareness as much as defect finding; understanding the change is the central challenge
  • Lisanne Bainbridge (1983). Ironies of Automation. Automatica, 19(6), 775–779.
    automation leaves the human the hardest parts and erodes the skills needed to intervene
  • Hamsa Bastani, Osbert Bastani, Alp Sungu et al. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences 122(26). Link
    an unguarded assistant improved practice and harmed learning once removed; a tutor that gave hints instead of answers largely avoided the harm
  • Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (editors) (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media.
    service-level objectives and error budgets; incident management; postmortem culture
  • Erik Brynjolfsson, Bharat Chandar, Ruyu Chen (2025). Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence. Stanford Digital Economy Lab working paper (first released August 2025; revised 2026). Link
    early-career workers in AI-exposed occupations falling behind their less exposed peers in employment; a working paper
  • Erik Brynjolfsson, Danielle Li, Lindsey Raymond (2025). Generative AI at Work. Quarterly Journal of Economics 140(2). Link
    an assistant raised customer-support productivity on average, most for novices and least for the most experienced
  • Nicholas Carlini, Florian Tramèr, Eric Wallace et al. (2021). Extracting Training Data from Large Language Models. Proceedings of the 30th USENIX Security Symposium. Link
    verbatim training data, including personal information, recovered from a language model by querying it
  • Lingjiao Chen, Matei Zaharia, James Zou (2024). How Is ChatGPT's Behavior Changing Over Time?. Harvard Data Science Review 6(2). Link
    the behaviour of the same named model service changed substantially within three months
  • Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick et al. (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science 37(2). Link
    AI helped consultants on tasks inside its capability and made them worse on a task outside it
  • DORA (Google Cloud) (2024). Accelerate State of DevOps Report 2024. Google Cloud. Link
    survey-based estimates; higher AI adoption associated with slightly lower delivery throughput, lower delivery stability, and faster code review
  • Tyna Eloundou, Sam Manning, Pamela Mishkin, Daniel Rock (2024). GPTs are GPTs: Labor market impact potential of LLMs. Science 384(6702). Link
    estimates of task exposure to language models; potential, not observed displacement
  • European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L, 12 July 2024. Link
    risk-based obligations; Annex III point 4 lists recruitment and selection, and decisions on promotion, termination and performance, as high-risk uses
  • Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck and Jenna Butler (2021). The SPACE of Developer Productivity: There's More to It Than You Think. ACM Queue, 19(1).
    five dimensions of developer productivity; no single metric suffices
  • Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec 2023). Link
    instructions planted in retrieved content can steer a model-integrated application
  • Anders Humlum, Emilie Vestergaard (2025). Large Language Models, Small Labor Market Effects. Becker Friedman Institute Working Paper 2025-56. Link
    Danish administrative data; precise null effects of AI chatbots on earnings and recorded hours, ruling out effects larger than 2% two years on
  • Chip Huyen (2025). AI Engineering: Building Applications with Foundation Models. O'Reilly Media.
    evaluation-driven development; choosing between prompting, retrieval and fine-tuning
  • Chip Huyen (2022). Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications. O'Reilly Media.
    data distribution shifts and monitoring in production
  • ISO/IEC (2023). ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system. International Organization for Standardization. Link
    requirements for an AI management system, on the clause structure of ISO/IEC 27001
  • Hao-Ping (Hank) Lee, Advait Sarkar, Lev Tankelevitch et al. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. Proceedings of CHI 2025. Link
    self-reported: more confidence in the tool went with less critical thinking, more self-confidence with more
  • Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). Link
    retrieval from a document index made generated answers more specific and factual, and traceable to sources
  • Nelson F. Liu, Kevin Lin, John Hewitt et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12. Link
    models use information at the start and end of a long context better than information in the middle
  • Rita Gunther McGrath, Ian C. MacMillan (1995). Discovery-Driven Planning. Harvard Business Review, July–August 1995. Link
    plan a venture by testing its assumptions at checkpoints, not by trusting a forecast
  • Joel Becker, Nate Rush, Elizabeth Barnes and David Rein (METR) (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR (Model Evaluation and Threat Research), July 2025.
    randomised controlled trial; experienced developers in their own repositories were slower with AI tools while believing they were faster
  • British Columbia Civil Resolution Tribunal (2024). Moffatt v. Air Canada, 2024 BCCRT 149. Civil Resolution Tribunal (small claims), decision of 14 February 2024. Link
    a company was held responsible for its website chatbot misstating a fare policy; a non-binding small-claims decision
  • Arvind Narayanan, Sayash Kapoor (2024). AI Snake Oil. Princeton University Press.
    how to tell inflated AI claims from real capability, especially in prediction and hiring
  • UK National Cyber Security Centre, US Cybersecurity and Infrastructure Security Agency et al. (2023). Guidelines for secure AI system development. NCSC, November 2023. Link
    secure design, development, deployment, and operation and maintenance of AI systems
  • National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Link
    four functions: govern, map, measure, manage
  • Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Proceedings of EMNLP 2024. Link
    retrieval outperformed unsupervised fine-tuning for adding knowledge; the comparison does not cover supervised fine-tuning
  • OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications 2025. OWASP Foundation, released November 2024. Link
    names for LLM risks, including excessive agency (LLM06) and unbounded consumption (LLM10)
  • Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt and Ramesh Karri (2022). Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. IEEE Symposium on Security and Privacy.
    a substantial share of assistant-generated code in security-relevant scenarios contained known vulnerability classes
  • Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh (2023). Do Users Write More Insecure Code with AI Assistants?. Proceedings of the ACM Conference on Computer and Communications Security (CCS 2023). Link
    participants with an assistant wrote less secure code and were more likely to believe it secure
  • Goran Petrović, Marko Ivanković (2018). State of Mutation Testing at Google. Proceedings of ICSE-SEIP 2018. Link
    mutation results on the changed lines, shown in code review at scale
  • Yangjun Ruan, Honghua Dong, Andrew Wang et al. (2024). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. International Conference on Learning Representations (ICLR 2024). Link
    tool-using agents took risky actions in a substantial share of high-stakes test cases; emulated tools, 2023 models
  • Max Schäfer, Sarah Nadi, Aryaz Eghbali, Frank Tip (2024). An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50(1). Link
    generated tests reached substantial coverage; coverage is not fault detection
  • Frank L. Schmidt and John E. Hunter (1998). The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings. Psychological Bulletin, 124(2), 262–274.
    meta-analysis of selection-method validity; structured interviews among the strongest predictors
  • D. Sculley, Gary Holt, Daniel Golovin et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NIPS 2015). Link
    the model is a small part of the system; data dependencies, feedback loops and changes in the outside world carry the cost
  • Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein, Aditya G. Parameswaran (2024). "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. Proceedings of the ACM on Human-Computer Interaction 8 (CSCW1). Link
    operating models is a continual loop of validation, versioning and monitoring
  • Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, Ian Arawjo (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Proceedings of UIST 2024. Link
    criteria drift, people define their grading criteria partly by grading outputs
  • Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. Link
    a strong model judge agreed with human preferences about as often as humans agree with each other, excluding ties, and showed position and verbosity bias