Patrick Queiroz Da Silva

Patrick Queiroz Da Silva

Ph.D. Student in CSE @ OSU

Talks and presentations

ScholarEval: Research Idea Evaluation Grounded in Literature

August 05, 2025

Invited, AI2 (Allen Institute For Artificial Intelligence), Seattle, Washington, United States

The growing capabilities of large language models have led to their increased adoption across the scientific lifecycle, spanning different stages from idea conception to experiment execution, manuscript writing, and peer review. Recent interest in using AI for scientific hypothesis generation has shown promising results, with some studies demonstrating that AI-generated ideas can score higher than human-generated ones in terms of novelty and excitement. However, many scientific hypotheses generated by LLMs tend to yield poor execution results, leading to wasted resources, particularly in resource-intensive fields requiring considerable computation or wet-lab experiments. An integral missing component of the AI-assisted scientific lifecycle is rigorous idea evaluation to prioritize the most promising ideas for execution. To address this gap, we present our ongoing work on ScholarEval, a multi-disciplinary research idea evaluation system grounded in literature. ScholarEval evaluates research ideas along two key dimensions: soundness and contribution, generating comprehensive idea reviews accompanied by citations and scores. We aim to release ScholarEval as an open tool for scientists to evaluate both human and AI-generated research ideas against the current literature, thereby improving research idea refinement and resource allocation. 

ACL Panel: Generalisation of NLP models

July 30, 2025

Panel, Association for Computational Linguistics, Vienna, Austria

At the Association for Computational Linguistics 2025 Conference, 25 individuals were selected from over 3,000 accepted papers to participate in 5 themed panels. I am grateful to have been given the opportunity to speak on the panel covering generalization. TLDR: The generalization of interpretability-based steering methods is at an inflection point. As a community, we need to place strong emphasis on methods-reliability evals if we care about long-term impact.

Steering off Course: Reliability Challenges in Steering Language Models

April 08, 2025

Master's Thesis, The Ohio State University, Columbus, Ohio, United States

The number of AI publications has nearly tripled from 2010 to 2022 (https://hai.stanford.edu/ai-index). This unprecedented rate of growth is leading to many great advancements, but the speed of development comes with a cost. As researchers scramble to push benchmarks and discover new capabilities, many fundamental scientific questions are glossed over. This pattern has contributed to a growing blind spot in the robustness of interpretability techniques for large language models. One such example is “steering”, which has gained traction as an interpretable and lightweight alternative to model training. We systematically examine three prominent steering methods—DoLa, function vectors, and task vectors. In contrast to the original studies, which evaluated a handful of models, we test up to 36 models belonging to 14 families with sizes ranging from 1.5B to 70B parameters. Our experiments reveal substantial variability in the effectiveness of the steering approaches, with a large number of models showing no improvement and at times degradation in steering performance. Our analysis reveals fundamental flaws in the assumptions underlying these methods, challenging their reliability as scalable steering solutions.