[Review] Automatic Detection of Java Cryptographic API Misuses: Are We There Yet?

Link here

A large study of Java cryptographic API misuse.

Two main contributions are made:

1. evaluate the effectiveness of existing cryptographic API misuse detection tools.
2. conduct a study with the developers, measuring the real-world performance of detectors.

Introduction:

JCA (Java Cryptography Architecture), JSSE (Java Secure Socket Extension).

Java cryptographic API misuses are common, which may cause a extensive security problems.

13 Java types frequently mentioned in the API-misuse patterns.

Implementation:

6 existing tools: CogniCrypt, CryptoGuard, CryptoTutor, FindSecBugs, SonarQube, and Xanitizer.

3 benchmarks: CryptoBench, MUBench, and OWASP Benchmark.

Evaluation criteria: precision, recall, F-score rates, and runtime overhead.

• RQ1: How are current tools designed to detect cryptographic API misuses?

• RQ2: How effectively do current tools work to locate cryptographic API misuses?

• RQ3: How do developers perceive the usefulness of tools’ outputs?

For RQ1:

• existing tools are different in terms of their availability, input formats, pattern representations, pattern-matching strategies, and outputs.
• most tools represent patterns as built-in rules, conduct inter-procedural analysis, and report detected API misuses as outputs.

For RQ2:

• the measured time costs imply that given hundreds of programs to scan, the experimented tools usually respond within six hours (18,000 seconds).
• no tool consistently worked best. However, CogniCrypt, CryptoGuard, and Xanitizer always outperformed SonarQube, probably due to their sophisticated inter-procedural analysis and larger pattern sets.

For RQ3:

• developers are usually negative towards the reported security-API misuses.
• reasons for this phenomenon:
  • Incomplete fixing suggestion: the fixing guidance offered by the tool is not complete, extra fixing efforts were needed.
  • Complex repair procedures: some repair process are challenging and time consuming, and developers have almost zero tool assistance for non-code artifact configuration.
• developers need tools to provide more detailed suggestions on repairing edits and non-code artifact configuration.
• some PRs(pull requests) were rejected, main because:
  • No exploit demo: actual security attacks to demonstrate the security exploit are required.
  • False positives without actual security impact: some misuse may cause no damage to the system.
• developers need tools to demonstrate security exploits of vulnerabilities, and to skip issues located in test cases, archived code, and security-irrelevant context.

Future work:

• some new methods to detect the misuse, other than the traditional static analysis.
• mix the use of different existing tools to achieve a better performance.
• attach the tools with some other functions, like security exploits of vulnerabilities.