A 360 review of AI agent benchmarks
Research
Kim Martineau
Natural Language Processing
Much of the information that can help transform enterprises is locked away in text, like documents, tables, and charts. We’re building advanced AI systems that can parse vast bodies of text to help unlock that data, but also ones flexible enough to be applied to any language problem.
Our work
The simplest protocol for AI agents to work together
ResearchKim MartineauAI is changing how we work — is it time to change how we credit AI’s involvement?
ResearchKim MartineauIBM crossed a transformer with an SSM and got ‘Bamba’
NewsKim MartineauA new kind of adapter helps LLMs get their words out faster
ResearchKim MartineauIBM’s safety checkers top a new AI benchmark
NewsKim Martineau- See more of our work on Natural Language Processing
Publications
ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages
- Mehant Kammakomati
- Sameer Pimparkhede
- et al.
- 2025
- ACL 2025
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
- Rickard Gabrielsson
- Jiacheng Zhu
- et al.
- 2025
- ICML 2025
Regress, Don't Guess: A Regression-like Loss on Number Tokens for Language Models
- Jonas Zausinger
- Lars Pennig
- et al.
- 2025
- ICML 2025
DiReC: Enhancing VHDL Code Generation and Summarization with Divide-Retrieve-Conquer Strategy
- Prashanth Vijayaraghavan
- Apoorva Nitsure
- et al.
- 2025
- DAC 2025
LogAn: An LLM-Based Log Analytics Tool with Causal Inferencing
- Pranjal Gupta
- Karan Bhukar
- et al.
- 2025
- ICPE 2025
Challenges and Remedies of Domain-Specific Classifiers as LLM Guardrails: Self-Harm as a Case Study
- Bing Zhang
- Guang-Jie Ren
- 2025
- NAACL 2025