ABRT plugin for Foreman

Discussion:

Martin Milata

2013-12-04 16:56:55 UTC

Hi!

I'm from the team that develops ABRT [1], the Automatic Bug Reporting Tool
that is currently used in Fedora and RH Enterprise Linux. ABRT can
detect several types of application problems (e.g. binary segfaults,
uncaught python exceptions, ...), collect related data, and more or less
automatically report them somewhere.

For the detected crashes, ABRT can send something called uReport [2], a
small, machine-friendly report that doesn't necessarily contain enough
information to fix the underlying problem but can be useful to determine
whether similar problem happened before and to maintain crash
statistics. These reports can be collected by ABRT server (called "FAF"
in some places) - Fedora users send them to the instance located here
[3].

What we would like to do is to provide a Foreman plugin that would
associate to every Foreman-managed machine the uReports the machine
sent. The reports would then be displayed in the web interface so that
the administrator can see if something bad is going on on their machine.

We're not familiar with Foreman and still trying to figure out how would
this integration work. After a quick discussion with Foreman guys in Red
Hat Brno office, we've identified two design problems:

1. How should Foreman and ABRT server interact
----------------------------------------------

We were able to come up with two scenarios:

A) uReports are collected by ABRT server deployed by the administrator.
Upon receiving a report, the ABRT server notifies Foreman (or Foreman
can periodically ask the ABRT server for new reports). Foreman
communicates with ABRT server using some kind of REST API.

* machine authentication has to be handled by ABRT server
* ABRT server has to provide suitable API
* not necessary to duplicate ABRT server code in Foreman

B) uReports are collected by Foreman, or some kind of proxy written for
this purpose. The reports can be browsed in Foreman and can be
forwarded to ABRT server instance, either automatically or after the
administrator manually accepts the report. The report can be
forwarded to ABRT server run by the administrator, or it can be
forwarded to the instance managed by Fedora/Red Hat/etc.

* Foreman handles machine authentication
* administrator can benefit from this without deploying their own ABRT
server
* subset of the ABRT server functionality would have to be implemented
by the Foreman plugin/proxy

2. Machine authentication
-------------------------

uReports were originally designed to allow anonymous reporting, mainly
for Fedora users. They only contain data that are not considered
sensitive and we currently have no way to find out where did an uReport
come from.

While we could just add an item containing e.g. FQDN to the uReport,
such information can be easily spoofed. Can we take advantage of the
fact that there already exists authentication between the managed
machines and Foreman (or Puppet?)?

I'll be grateful for your thoughts on this subject.
Cheers,
Martin

[1] https://github.com/abrt
[2] https://github.com/abrt/faf/wiki/uReport#ureport2
[3] https://retrace.fedoraproject.org/faf/summary/

Martin Milata

2013-12-06 16:02:07 UTC

Permalink

On Thu, Dec 05, 2013 at 15:37:42 +0200, Ohad Levy wrote:
[...]

Post by Martin Milata
A) uReports are collected by ABRT server deployed by the administrator.
Upon receiving a report, the ABRT server notifies Foreman (or Foreman
can periodically ask the ABRT server for new reports). Foreman
communicates with ABRT server using some kind of REST API.
* machine authentication has to be handled by ABRT server
* ABRT server has to provide suitable API
* not necessary to duplicate ABRT server code in Foreman

Does it make sense to have multiple abrt servers? does it make sense to tie
these into foreman proxies (e.g. when having disconnected networks?)

I don't think it does. One of the main features of the server is
grouping the reports that were probably caused by the same bug. I'd say
it's best to have just one server.

Post by Martin Milata
B) uReports are collected by Foreman, or some kind of proxy written for
this purpose. The reports can be browsed in Foreman and can be
forwarded to ABRT server instance, either automatically or after the
administrator manually accepts the report. The report can be
forwarded to ABRT server run by the administrator, or it can be
forwarded to the instance managed by Fedora/Red Hat/etc.
* Foreman handles machine authentication
* administrator can benefit from this without deploying their own ABRT
server
* subset of the ABRT server functionality would have to be implemented
by the Foreman plugin/proxy

Which level of complexity are we talking about here? does it make sense to
rewrite the abrt server? (or is it only a subset of the functionality?)

The server is quite complex as it does e.g. the report clustering or
resolving function names in stacktraces. I don't think rewriting it
entirely is reasonable. However, I can imagine that implementing a tiny
subset that would allow the administrator to see that a crash happened
in program X (from package Y) Z times in some timeframe would still be
useful.

Post by Martin Milata
2. Machine authentication
-------------------------
uReports were originally designed to allow anonymous reporting, mainly
for Fedora users. They only contain data that are not considered
sensitive and we currently have no way to find out where did an uReport
come from.
While we could just add an item containing e.g. FQDN to the uReport,
such information can be easily spoofed. Can we take advantage of the
fact that there already exists authentication between the managed
machines and Foreman (or Puppet?)?

I would assume in this case, you would want your systems to be known, we
could utilize the system facts, or uuid to identify it.

I think that without this assumption there wouldn't be many reasons to
integrate abrt server with foreman, as opposed to just running the
standalone abrt server somewhere on the network.

Thanks for the response,
Martin

Martin Milata

2013-12-06 16:20:40 UTC

Permalink

Post by Martin Milata
Hi!

Hey,

We have nice Puppet based installer, so this looks like viable option to
me as we can setup another component.

Post by Martin Milata
B) uReports are collected by Foreman, or some kind of proxy written for

I prefer A, this is solely my opinion.

Post by Martin Milata
While we could just add an item containing e.g. FQDN to the uReport,
such information can be easily spoofed. Can we take advantage of the
fact that there already exists authentication between the managed
machines and Foreman (or Puppet?)?

Someone correct me if I am wrong, but we are deploying Puppet client
certificate during provisioning phase which is being signed by Puppet CA
authority. That means each Foreman-managed machine has a client
certificate which could be re-used for other things. It should not be
a problem to use Puppet CA to validate client certificates during ABRT
upload.
The key is to make sure ABRT server has access to the Puppet CA
certificate (and key).

I wonder if it would be possible to use these certificates without major
changes to Puppet. Or, whether the benefits of having authenticated
problem reports outweigh the risks of sharing the puppet certificates
with another component.

Thanks for the reply,
Martin

Lukas Zapletal

2013-12-07 10:39:31 UTC

Permalink

Post by Martin Milata
I wonder if it would be possible to use these certificates without major
changes to Puppet. Or, whether the benefits of having authenticated
problem reports outweigh the risks of sharing the puppet certificates
with another component.

Well the same way we deploy client certificate, we can also deploy
another ABRT certificate.

--
Later,

Lukas "lzap" Zapletal
irc: lzap #theforeman

Martin Milata

2014-01-20 15:24:59 UTC

Permalink

Post by Lukas Zapletal

Well the same way we deploy client certificate, we can also deploy
another ABRT certificate.

We just implemented the ability to use client-side SSL/TLS
authentication when sending uReports. This makes using Puppet
certificates easy as everything is handled by libcurl and it appears
that SELinux already allows us to access them.

If I understand correctly the plugin has to provide a smart proxy to
accept the uReports since the managed hosts are not required to have
connectivity to the main Foreman server. The idea is that the proxy
receives the uReports via client-authenticated https where the Puppet CA
is used to validate the client's certificate. Do you see any problem
with this?

We'd like to make the ureport client configuration easy so that the path
to the Puppet certificate doesn't have to be specified explicitly. I'm
not familiar with Puppet so I'd like to ask if we can expect the client
certificate to be in /var/lib/puppet/ssl/certs/`facter fqdn`.pem
and the key in /var/lib/puppet/ssl/private_keys/`facter fqdn`.pem ?

Thank you,
Martin Milata