Ingress
Host data for processing by the Insights services (usually collected by the Insights Client) is received by the Ingress service. The data is packaged and received as tarball archives (gzipped tar). The archives are processed by an upload processor before they are dispatched for analysis.
See also Architecture Overview and Insights Client pages.
Archive Ingestion
Archives with host data are sent via HTTP POST to an Ingress API /api/ingress/v1/upload. Clients on hosts utilize certificate authentication and post the archives through Foreman via:
Archives are routed to a specific Insights service based on their MIME type:
For example, data collected by Insights Core or the Insights Client uses this MIME type:
The on-premises deployment of the Ingress service uses a disk volume to store archives temporarily. The temporary directory is mounted as a volume, typically from /var/tmp/insights-archives. How long archives remain on disk is controlled by a tempfiles.d systemd configuration.
Ingress provides an API to download archives internally for services that process them further. Once an archive is received and stored, it is announced on a Kafka topic:
The event message includes the service type (based on the archive MIME type), host and organization information, and an internal URL for downloading the archive. See the README of insights-ingress-go on GitHub.
Upload Processors
Upload processors validate the archives, transform them, and send them to the appropriate service for further analysis via Kafka topics.
There are two main upload processors in Insights:
- Puptoo – processes data from Insights Core collectors (part of Insights Client)
- Yuptoo – processes reports from Subscription Manager (generated by Foreman, RHCloud plugin)
Puptoo processes archives that are targeted (by MIME type) for advisor, compliance, and malware-detection. Puptoo announces successful processing to Host Inventory via the topic:
Yuptoo processes uploads of type qpc and sends them to Host Inventory through the topic:
Archived reports for Yuptoo are generated by the Foreman RHCloud plugin with bulk host data from Subscription Manager.
Whilst the Insights Client is the primary archive generator, Foreman reports processed by Yuptoo provide a secondary source of host data. Data from Foreman reports is used, for example, by the Vulnerability service to react quickly to host changes, such as installed packages, without waiting for a periodic daily run of the Insights Client.