Ingress
Host data for processing by the Insights services (usually collected by the Insights Client) is received by the Ingress service. The data is packaged and received as tarball archives (gzipped tar). The archives are processed by an upload processor before they are dispatched for analysis.
See also Architecture Overview and Insights Client pages.
Archive Ingestion
Archives with host data are sent via HTTP POST to an Ingress API /api/ingress/v1/upload. Clients on hosts utilize certificate authentication and post the archives through Foreman via:
Archives are routed to a specific Insights service based on their MIME type:
For example, data collected by Insights Core or the Insights Client uses this MIME type:
Once an archive is received and stored, it is announced on a Kafka topic:
The event message includes the service type (based on the archive MIME type), host and organization information, and an internal URL for downloading the archive. See the README of insights-ingress-go on GitHub.
Archive Storage
In cloud deployments, Ingress stages archives to S3-compatible object storage and shares presigned URLs with downstream services. On premises, S3 is not available — Ingress uses local filesystem storage instead (see Minimal Dependencies).
Archives are written as flat files to a configured directory, typically /var/tmp/insights-archives. Each file is named after the request ID (from the x-rh-insights-request-id header) with a .tar.gz extension:
When filesystem storage is active, Ingress exposes an additional HTTP endpoint at /download/{requestID} that serves stored archives. The download URL published in the Kafka announcement points to this endpoint, so downstream processors (Puptoo, Yuptoo) fetch archives directly from Ingress over HTTP.
Cleanup of stored archives is handled by systemd's tmpfiles.d mechanism (see Temporary Data Volume Pruning).
Configuration
The storage mode and related settings are controlled by environment variables (prefixed with INGRESS_):
Upload Processors
Upload processors validate the archives, transform them, and send them to the appropriate service for further analysis via Kafka topics.
There are two main upload processors in Insights:
- Puptoo – processes data from Insights Core collectors (part of Insights Client)
- Yuptoo – processes reports from Subscription Manager (generated by Foreman, RHCloud plugin)
Puptoo processes archives that are targeted (by MIME type) for advisor, compliance, and malware-detection. Puptoo announces successful processing to Host Inventory via the topic:
Yuptoo processes uploads of type qpc and sends them to Host Inventory through the topic:
Archived reports for Yuptoo are generated by the Foreman RHCloud plugin with bulk host data from Subscription Manager.
Whilst the Insights Client is the primary archive generator, Foreman reports processed by Yuptoo provide a secondary source of host data. Data from Foreman reports is used, for example, by the Vulnerability service to react quickly to host changes, such as installed packages, without waiting for a periodic daily run of the Insights Client.