Jie He

Large Language Models (LLMs) can enhance their capabilities as AI assistants by integrating external tools, allowing them to access a wider range of information. We present GenTool, a novel training framework addressing two dimensions: Zero-to-One Generalization and Weak-to-Strong Generalization. Through extensive experiments across four generalization scenarios, our method significantly enhances tool-usage capabilities of LLMs ranging from 1B to 8B parameters, achieving performance that surpasses GPT-4o.

We conduct a comprehensive evaluation of prompting open-source LLMs on graph-to-text generation tasks. We propose a novel diversity-difficulty-based few-shot sample selection method and introduce PlanGTG, a new graph-to-text dataset annotated with reordering and attribution sub-tasks. Extensive evaluations demonstrate significant improvements in generated text quality from both few-shot and fine-tuning perspectives.

We propose MiCEval, a framework to assess the correctness of reasoning chains in Multimodal Chain of Thought (MCoT) by evaluating the quality of both the description and each reasoning step. MiCEval is built upon a fine-grained dataset with annotations rating each step according to correctness, relevance, and informativeness. Experiments on four MLLMs show step-wise evaluations align more closely with human judgments.

We study the robustness of retrieval-augmented ICL against adversarial attacks. Retrieval-augmented models enhance robustness against test sample attacks (4.87% ASR reduction) but exhibit overconfidence in demonstrations (2% ASR increase). We introduce DARD, a training-free adversarial defence method, achieving a 15% reduction in ASR over baselines.

We study Parameter-Efficient Fine-Tuning (PEFT) methods for MLLMs, evaluating four PEFT methods on seven datasets. We show that the adapter is the best-performing PEFT method, and fine-tuning connector layers leads to improved performance in most MLLMs.

We propose to transform the downstream multiple choice QA task into a simpler binary classification task by ranking all candidate answers according to their reasonableness. We convert knowledge graph triples into reasonable and unreasonable texts for training. Extensive results show effectiveness on various QA benchmarks, with our approach being less data hungry than existing methods.

We propose TGEA, using carefully selected prompts to guide GPT-2 to generate candidate sentences, selecting 47K for error annotation. We create an error taxonomy covering 24 types of errors and propose a series of automatic diagnosis tasks including error detection, error type classification, associated span detection, and error rationale generation.

We present a test suite to evaluate the commonsense reasoning capability of neural machine translation, covering lexical and syntactic ambiguity that requires commonsense knowledge to resolve. We manually create 1,200 triples and demonstrate that neural machine translation performs poorly on commonsense reasoning.