correlating static analysis results with known crashes

[Martin Milata]

Hello,

I've implemented a proof-of-concept of an analysis that tries to pair
static analysis results with known crashes based on the source code
locations, as outlined in [1].

The code extends David Malcolm's mock-with-analysis and is available at
[2]. The machinery for generating static analysis results is unchanged
apart from a few fixes needed for it to run on Fedora 19. The script
make-simple-report.py was extended to accept second argument with crash
reports from FAF server, and the matching crashes are referenced in the
generated reports. There is currently no way to obtain the file with
crashes automatically, I got it from the server administrator.

I ran the analysis on following packages:
* tracker-0.16.2-1.fc19
* evolution-3.6.4-3.fc18
* gnome-shell-3.6.3.1-1.fc18
* nautilus-3.6.3-4.fc18
* python-2.7.3-13.fc18
* rhythmbox-2.98-4.fc18

Tracker was chosen arbitrarily, the rest of the builds are those that
have the highest number of distinct crashes. The results can be viewed
at [3] and given that they were obtained from packages with the highest
number of collected crashes, they don't seem to be very encouraging.
There are only three [4,5,6] matches that are not obvious false
positives. All the data needed to reproduce this should be available at
[7].

There are two main causes of false positives:
* The code considers all static analysis results, not only those from
tests for behaviour that would result in a crash at runtime.
* It considers all stack frames in a crash, not just the topmost one.

As a side note, all three matches come from the clang static analyzer,
which for some reason fails for quite a lot of source files.

What do you think?

Thanks,
Martin


[1] https://lists.fedoraproject.org/pipermail/firehose-devel/2013-October/000065.html
[2] https://github.com/mmilata/mock-with-analysis/tree/crash-correlation
[3] http://mmilata.fedorapeople.org/firehose-crash-correlation/
[4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5223
[5] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5848
[6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources/71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97.html#file-71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97-line-1171
[7] http://mmilata.fedorapeople.org/firehose-crash-correlation.tar.xz

[Martin Milata]

I uploaded the clang-analyzer-generated html reports for the three
"interesting" cases that the script found and took a further look at
them.

* nautilus 1 [1], clang-analyzer report [2]

The trace from the static analyzer consists of
nautilus_file_operations_copy_move calling nautilus_file_operations_move
which then segfaults. This agrees with the backtraces. Unfortunately
there is no BZ ticket associated probably due to too few people affected
by this bug

* nautilus 2 [3], clang-analyzer report [4]

Only nautilus_file_operations_copy_move is in the static analyzer trace.
There's bugzilla ticket [5] with full backtrace corresponding to this
problem.

* python [6], clang-analyzer report [7]

The trace consists of PyObject_Unicode calling PyObject_GetAttr, which
is not the case of the linked backtrace, making this pair a false
positive. The trace from clang-analyzer describes a real bug though, one
that has been already fixed [8][9].

Didn't know clang-analyzer can do inter-procedural analysis, that's
nice.


[1] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5223
[2] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-build/report-eEHeBD.html#Path1

[3] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5848
[4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-build/report-YLMHRs.html#Path1
[5] https://bugzilla.redhat.com/show_bug.cgi?id=860109

[6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources/71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97.html#file-71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97-line-1171
[7] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/scan-build/report-cp0FYq.html#Path1
[8] http://bugs.python.org/issue16839
[9] http://hg.python.org/cpython/rev/0012d4f0ca59

[Paul Tagliamonte]

Post by Martin Milata
I uploaded the clang-analyzer-generated html reports for the three
"interesting" cases that the script found and took a further look at
them.
* nautilus 1 [1], clang-analyzer report [2]
The trace from the static analyzer consists of
nautilus_file_operations_copy_move calling nautilus_file_operations_move
which then segfaults. This agrees with the backtraces. Unfortunately
there is no BZ ticket associated probably due to too few people affected
by this bug
* nautilus 2 [3], clang-analyzer report [4]
Only nautilus_file_operations_copy_move is in the static analyzer trace.
There's bugzilla ticket [5] with full backtrace corresponding to this
problem.
* python [6], clang-analyzer report [7]
The trace consists of PyObject_Unicode calling PyObject_GetAttr, which
is not the case of the linked backtrace, making this pair a false
positive. The trace from clang-analyzer describes a real bug though, one
that has been already fixed [8][9].
Didn't know clang-analyzer can do inter-procedural analysis, that's
nice.

Thrilling stuff, nice work!

I'll soon have a corpus of checks being run against Debian packages,
I'll be sure to forward you data points (if y'all have the same
source/version pair in Fedoraland)

Keep up the great work,
Paul

[Martin Milata]

Post by Paul Tagliamonte
Thrilling stuff, nice work!
I'll soon have a corpus of checks being run against Debian packages,
I'll be sure to forward you data points (if y'all have the same
source/version pair in Fedoraland)

Thanks! It would be interesting to run the analysis on those, though the
possible differences between Debian and Fedora sources could pose a
problem. E.g. a patch in one package and not the other might cause the
line numbers to disagree. I have no idea how often this is the case.

Martin

[David Malcolm]

Post by Martin Milata

Post by Paul Tagliamonte
Thrilling stuff, nice work!
I'll soon have a corpus of checks being run against Debian packages,
I'll be sure to forward you data points (if y'all have the same
source/version pair in Fedoraland)

Thanks! It would be interesting to run the analysis on those, though the
possible differences between Debian and Fedora sources could pose a
problem. E.g. a patch in one package and not the other might cause the
line numbers to disagree. I have no idea how often this is the case.

That in itself might be something we could track using firehose,
perhaps? i.e. have an <info> element that says that the code is
patched downstream by a particular distribution. Then the UI can render
those elements (though which version of the source would you render in
such a situation), and one can run a query showing patches across
multiple distros and packages.

(Not sure if this is a good idea, but I thought I'd share it)

Dave

[Stefano Zacchiroli]

Post by David Malcolm
That in itself might be something we could track using firehose,
perhaps? i.e. have an <info> element that says that the code is
patched downstream by a particular distribution. Then the UI can
render those elements (though which version of the source would you
render in such a situation), and one can run a query showing patches
across multiple distros and packages.

Uhm, so the broader need here is the ability to correlate different
distro-specific versions with one another or, in fact, to the respective
upstream version. We can do that via external databases, but it would
add a pretty heavy infrastructure dependency.

IMO it would be better to pursue one of the following two solutions (or
even both):

- add a new sub-element to <sut> which mentions the *upstream* version;
once we have that we can correlate reports from different distros via
the upstream version (if there are significant differences, that
should come from the distro-specific patching)

- add a new <context> or <excerpt> sub-element to failure/info/etc. that
can be used to add snippets of code around the location the static
analysis tool is pointing at. The idea of this is the same of contexts
for textual diffs: by comparing them we will be able to understand if
we're talking about the same code or, due to patched, significantly
different parts of it.

Of course it's an approximated solution, as the failure might descend
from patches far far away in the code base, but if it works for diff,
I think it'd be good enough for us as well. (And if we also have the
upstream version, we can always lookup the distro-specific patches by
external means and compare those.)

Just my 0.02 EUR,
Cheers.