[Review] Assisting Static Analysis with Large Language Models: A ChatGPT Experiment

Link

The paper demonstrates the effectiveness of LLM in static analysis.

The most important thing of this paper is the task division and the workflow design. First we need to figure out what the LLM is good at, and assign such tasks to it. What’s more, we need to care about the design of the workflow, which could significantly affect the final result.

Background

Traditional static analysis tools have some shortages. Embedding LLM into the toolchain can help the analysis.

In this paper, Use Before Initialization (UBI) bugs are chosen as the example.

UBITect, which is a tool for UBI bugs, has some shortcomings in detecting, and may discord some cases. LLM can help determine whether these bugs are true bugs.

Implementation

• Chain-of-Thought: add “think step by step” in the prompt. (not obviously used in the paper)
• Task Decomposition: break down the problem into multiple steps and instruct LLM to complete smaller ones. When we need a structured output, always initiate a new request at the end of the conversation and prompt LLM to conclude with JSON format separately.
• Progressive Prompt: Iterative prompting. “If you experience uncertainty due to insucient function denitions, please indicate the required functions.”