[Review] Testing Database Engines via Pivoted Query Synthesis

Link here

This paper aims to detect the logic bugs in the DBMSs. In this paper, logic bugs are defined as bugs that cause a query to return an incorrect result without crashing the DBMS. It randomly selects a row from the table(called pivot row) and syntheses a query whose result should contain the selected row, and then sends the query to the DBMS. By checking if the pivot row is contained in the result, we will know if there is a logic bug.

Motivation:

Logic bugs in DBMSs are hard to find. While former logic bugs detector RAGS applying differential testing to itself, there are still a lot of problems because of $DBMSs'$ $dialects^{1}$ and the $common$ $bugs^{2}$. So SQLancer is proposed to tackle this problem.

[1]: Different DBMSs have different ways of implementation and unique grammar towards the same SQL query.

[2]: Different DBMSs may have the same bug, which disables differential testing.

Implementation:

• Randomly generate tables and rows.
• Randomly select a row from each table(the pivot row may cross several tables).
• Randomly generate a AST based on the database’s schema(the column names and types).
• Correct the result of the generated AST(keep still with TRUE and add a NOT with FALSE).
• Transform the AST into the SQL query.
• Send the query to DBMSs to test(we can use INTERSECT or IN provided by the DBMS to help our test).

Evaluation:

• Setup: A laptop with a 6-core Intel i7-8850H CPU at 2.60 GHz and 32 GB of memory running Ubuntu 19.04.
• Interestingly, authors say there is no other tools to compare with except RAGS, which was proposed more than 20 years ago and was with low efficiency. So in the evaluation section, authors only mention the implementation effort and the coverage, as well as introduce the bugs they found in the test.
• Implementation effort: Compares the LOC(line of code) of SQLancer and DBMSs(quite strange!).
• Coverage: The coverage is low, because they only tested the data-centric SQL statements.

Future work:

• Try to avoid duplicate problem(several queries trigger the same problem).
• Check the correct insertion or deletion of records, detect concurrency bugs, bugs related to transactions, or bugs in the access control layer of DBMSs.
• Test multiple rows.
• To solve the dialect problem of different DBMSs.