HTTP API Downloader

HTTP API Downloader allows you to serve feeds based on contents exposed by an HTTP API endpoint.

The main functionality is captured in the below diagram:

Image title — Overview of HTTP API Downloader

In the diagram you can see the main 2 components:

httpapi container is responsible for monitoring the configured HTTP API endpoint and synchronising the exposed contents to the local "feed storage"
webserver container is responsible for exposing the feed stored in "feed storage"

When deployed, HTTP API Downloader will expose a feed URL at the following path:

[Service / Ingress]/feed

Services created for Command Source are as follows:

HTTP: Port 8080 on Service httpapi-[Instance Set Name]
HTTPS (if enabled): Port 8443 on Service httpapi-[Instance Set Name]-https

Configuration Reference

Instance Sets

Instances of HTTP API Downloader can be defined under the root node httpAPIFeeds. Each instance set should be a key-value pair, with:

key: Name of the instance set
value: Dictionary holding the configuration of the instance set

httpAPIFeeds:
  myhttpapi:
    < HTTP API Downloader configuration >

In a minimal configuration you will need to specify at least the url of an API to mirror

For example:

httpAPIFeeds:
  myhttpapi:
    upstream:
      url: "https://my.upstream.api/api/v1/endpoint"
    service:
      type: NodePort

In the above example HTTP API Downloader will attempt to access the API at url and serve a feed with the corresponding contents. The corresponding Service will have type set to NodePort.

Parameters which can be used to further configure HTTP API Downloader instances for a specific instance set are shown in the below table.

Parameter	Type	Default	Description
`affinity`	`k8s:`Affinity		pod affinity (Kubernetes docs: Affinity and anti-affinity). If unset, a default anti-affinity is applied using `antiAffinityPreset` to spread pods across nodes
`agentLogLevel`	`string`	`"info"`	Verbosity of logging for the agent container. Available options: `"debug"` `"info"` `"warn"` `"error"`
`agentLogFormat`	`string`	`"text"`	Format of logging for the agent container. Available options: `"text"` `"json"`
`agentResources`	`k8s:`Resources	`limits: cpu: 250m memory: 256Mi`	Resources allocated to the agent container if `resourceDefaults` (global) is `true`
`antiAffinityPreset`	`string`	`"preferred"`	pod anti affinity preset. Available options: `"preferred"` `"required"`
`containerSecurityContext`	`k8s:`SecurityContext	`allowPrivilegeEscalation: false readOnlyRootFilesystem: true seccompProfile: type: RuntimeDefault capabilities: drop: - "ALL"`	SecurityContext applied to each container
`disableSticky`	`boolean`	`false`	If `true`, disable the operator-controlled selection of a primary pod to handle all inbound traffic
`expiry`	Feed Expiry	`{}`	Expiry configuration
`hostNetwork`	`boolean`	`false`	Use host networking for pods
`ingress`	`k8s:`IngressSpec	`{}`	Ingress configuration
`logFormat`	`string`	`"human"`	Format of logging. Available options: `"human"` `"json"`
`logLevel`	`string`	`"info"`	Level of logging. Available options: `"debug"` `"info"` `"warn"` `"error"`
`nodeSelector`	`k8s:`NodeSelector	`{}`	Kubernetes pod nodeSelector
`podAnnotations`	`k8s:`Annotations	`{}`	Annotations to be added to each pod
`podDisruptionBudget`	`k8s:`PodDisruptionBudgetSpec	`{}`	Spec of PodDisruptionBudget to be applied to deployment
`podLabels`	`k8s:`Labels	`{}`	Labels to be added to each pod
`podSecurityContext`	`k8s:`PodSecurityContext	`fsGroup: 953 runAsUser: 953 runAsGroup: 953 runAsNonRoot: true`	SecurityContext applied to each pod
`readyInterval`	`integer`	`5`	How often readiness of the HTTP API Downloader should be calculated in seconds
`replicas`	`integer`	`2`	Default number of replicas in a Deployment
`resources`	`k8s:`Resources	`limits: cpu: 1 memory: 2Gi`	Resources allocated to the httpapi container if `resourceDefaults` (global) is `true`
`service`	Service	`type: ClusterIP`	Service configuration
`serviceHTTPS`	Service	`type: ClusterIP`	HTTPS Service configuration Note: Only created when `tls.enabled = true`
`serviceLabels`	`k8s:`Labels	`{}`	Labels to be added to each service
`tls`	Inbound TLS	`enabled: false`	TLS configuration for inbound HTTPS traffic
`tolerations`	List of `k8s:`Tolerations	`[]`	Kubernetes pod Tolerations
`topologySpreadConstraints`	List of `k8s:`TopologySpreadConstraint	`[]`	Kubernetes pod topology spread constraints
`unreadyAfterNoSuccessPeriod`	`integer`	`0`	If HTTP API Downloader has not successfully ran for this many seconds, mark the pod as Unready. Disabled if 0 (default)
`upstream`	Upstream	`{}`	Upstream configuration
`users`	List of `string`	`["feeduser"]`	List of usernames for which a Secret containing the username and a generated password is created, which is given access to the feed. Note: The generated passwords are stored in base64 encoded form in the secret. Make sure you decode them
`userSecrets`	List of `string`	`[]`	List of names of Secrets containing `username` and `password` items which should act as credentials allowed to access the feed
`webserverHTTPExtra`	List of `string`	`[]`	Extra lines of NGINX configuration to be injected into the `server` directive for the HTTP traffic. Should not include a trailing `;`. For example: `webserverHTTPExtra: - "add_header X-HTTP-1 shortval" - "add_header X-HTTP-2 'longer value'"`
`webserverHTTPSExtra`	List of `string`	`[]`	Extra lines of NGINX configuration to be injected into the `server` directive for the HTTPS traffic. Should not include a trailing `;`. For example: `webserverHTTPExtra: - "add_header X-HTTPS-1 shortval" - "add_header X-HTTPS-2 'longer value'"`
`webserverResources`	`k8s:`Resources	`limits: cpu: 250m memory: 256Mi`	Resources allocated to the webserver container if `resourceDefaults` (global) is `true`
`webserverSecretName`	`string`		Name of a secret containing `username: base64_password` pairs to be granted access to the feed. `base64_password` should be the base64 encoded representation of the actual password Deprecated: Replaced by `userSecrets`
`webserverUsers`	List of `string`	`["csuser"]`	List of usernames for which a password should be generated and allowed access to the feed. By default a user named `csuser` will be created with a randomized password which can be obtained from the corresponding secret. Note: The generated passwords are stored in base64 encoded form in the secret. Make sure you decode them Deprecated: Replaced by `users`

Feed Expiry Configuration

Parameters to configure the expiration of feeds. For example:

httpAPIFeeds:
  myhttpapi:
    upstream:
      url: "https://my.upstream.api/api/v1/endpoint"
    expiry:
      snapshots:
        must_keep_count: 10
        max_keep_count: 20
      deltas:
        must_keep_count: 40
        max_keep_count: 100

The expiry node allows for configuration of expiration of snapshots and deltas, corresponding to the full snapshots & accompanying incremental deltas offered by the mirror. By default, the expiration mechanism is disabled.

Both snapshots and deltas can be configured with the following parameters:

Parameter	Type	Description
`max_keep_count`	`integer`	Remove the oldest updates if this value is configured and the current count exceeds this value. `must_` parameters supersede this.
`max_keep_interval`	`go:`DurationString	Remove updates if this value is configured and there are updates present which are older. `must_` parameters supersede this.
`must_keep_count`	`integer`	Minimum number of updates to retain
`must_keep_interval`	`go:`DurationString	Minimum age of an update before it can be considered for expiration

Inbound TLS

Parameters to configure TLS for inbound traffic. When enabled is set to true and a certificate source is configured, an additional Service with the name suffix -https will be created. An example:

httpAPIFeeds:
  myhttpapi:
    upstream:
      url: "https://my.upstream.api/api/v1/endpoint"

    tls:
      enabled: true
      certSecretName: my-httpapi-certificate

In the above example the certificate present in Secret my-httpapi-certificate will be attempted to be used to start a TLS-enabled listener.

Parameter	Type	Default	Description
`certSecretName`	`string`		Name of a Secret object containing a certificate (must contain the `tls.key`, `tls.crt` items)
`certManager`	`boolean`	`false`	Toggle to have a request created for Certmanager to provision a certificate. By default, this will request for a Certificate covering the following: - `httpapi-[Name of instance set]-https` - `httpapi-[Name of instance set]-https.[Namespace]` - `httpapi-[Name of instance set]-https.[Namespace].svc` Additional entries can be configured using `extraDNSNames`
`enabled`	`boolean`	`false`	Toggle to enable TLS If set to `true`, a `certSecretName` must be set or `certManager` must be set to `true` to ensure a valid certificate is available
`extraDNSNames`	List of `string`	`[]`	List of additional entries to be added to the Certificate requested from Certmanager
`issuerGroup`	`string`	`"cert-manager.io"`	Group to which issuer specified under `issuerKind` belongs Default value is inherited from the global `certManager` configuration
`issuerKind`	`string`	`"ClusterIssuer"`	Type of Certmanager issuer to request a Certificate from Default value is inherited from the global `certManager` configuration
`issuerName`	`string`	`""`	Name of the issuer from which to request a Certificate Default value is inherited from the global `certManager` configuration
`certSpecExtra`	CertificateSpec	`{}`	Extra configuration to be injected into the Certmanager Certificate object's `spec` field. Disallowed options: `"secretName"` `"commonName"` `"dnsNames"` `"issuerRef"` (These are configured automatically and/or via other options)
`certLabels`	`k8s:`Labels	`{}`	Extra labels for the Certmanager Certificate object
`certAnnotations`	`k8s:`Annotations	`{}`	Extra annotations for the Certmanager Certificate object

Service Configuration

Parameters to configure the service object for this deployment. For example:

httpAPIFeeds:
  myhttpapi:
    upstream:
      url: "https://my.upstream.api/api/v1/endpoint"
    service:
      type: LoadBalancer
      annotations:
        metallb.universe.tf/address-pool: name_of_pool

Parameter	Type	Default	Description
`allocateLoadBalancerNodePorts`	`boolean`	`true`	If true, services with type `LoadBalancer` automatically assign NodePorts. Can be set to `false` if the LoadBalancer provider does not rely on NodePorts
`annotations`	`k8s:`Annotations	`{}`	Annotations for the service
`clusterIP`	`string`		Static cluster IP, must be in the cluster's range of cluster IPs and not in use. Randomly assigned when not specified.
`clusterIPs`	List of `string`		List of static cluster IPs, must be in the cluster's range of cluster IPs and not in use.
`externalIPs`	List of `string`		List of IP addresses for which nodes in the cluster will also accept traffic for this service. These IPs are not managed by Kubernetes and must be user-defined on the cluster's nodes
`externalTrafficPolicy`	`string`	`Cluster`	Can be set to `Local` to let nodes distribute traffic received on one of the externally-facing addresses (`NodePort` and `LoadBalancer`) solely to endpoints on the node itself
`healthCheckNodePort`	`integer`		For services with type `LoadBalancer` and externalTrafficPolicy `Local` you can configure this value to choose a static port for the NodePort which external systems (LoadBalancer provider mainly) can use to determine which node holds endpoints for this service
`internalTrafficPolicy`	`string`	`Cluster`	Can be set to `Local` to let nodes distribute traffic received on the `ClusterIP` solely to endpoints on the node itself
`ipv4`	`boolean`	`false`	If true, force the Service to include support for IPv4, ignoring globally configured IP Family settings and/or cluster defaults. If `ipv4` is set to `true` and `ipv6` remains `false`, the result will be an `ipv4`-only SingleStack Service. If both are `false`, global settings and/or cluster defaults are used. If both are `true`, a PreferDualStack Service is created
`ipv6`	`boolean`	`false`	If true, force the Service to include support for IPv6, ignoring globally configured IP Family settings and/or cluster defaults. If `ipv6` is set to `true` and `ipv4` remains `false`, the result will be an `ipv6`-only SingleStack Service. If both are `false`, global settings and/or cluster defaults are used. If both are `true`, a PreferDualStack Service is created
`labels`	`k8s:`Labels	`{}`	Labels to be added to the service
`loadBalancerIP`	`string`		Deprecated Kubernetes feature, available for backwards compatibility: IP address to attempt to claim for use by this LoadBalancer. Replaced by `annotations` specific to each LoadBalancer provider
`loadBalancerSourceRanges`	List of `string`		If supported by the LoadBalancer provider, restrict traffic to this LoadBalancer to these ranges
`loadBalancerClass`	`string`		Used to select a non-default type of LoadBalancer class to ensure the appropriate LoadBalancer provisioner attempt to manage this LoadBalancer service
`publishNotReadyAddresses`	`boolean`	`false`	Service is populated with endpoints regardless of readiness state
`sessionAffinity`	`string`	`None`	Can be set to `ClientIP` to attempt to maintain session affinity.
`sessionAffinityConfig`	`k8s:`SessionAffinityConfig	`{}`	Configuration of session affinity
`type`	`string`	`ClusterIP`	Type of service. Available options: `"ClusterIP"` `"LoadBalancer"` `"NodePort"`

Upstream Configuration

Parameters to configure the upstream API from which to construct the feed. For example:

httpAPIFeeds:
  myhttpapi:
    upstream:
      url: "https://my.upstream.api/api/v1/endpoint"

Parameter	Type	Required	Default	Description
`apiKeyHeaderName`	`string`		`"X-API-Key"`	If `apiKeySecretName` is specified: name of the header added to each request towards the API containing the configured API key
`apiKeySecretKey`	`string`		`"api_key"`	If `apiKeySecretName` is specified: name of the item inside the Secret which holds the API key
`apiKeySecretName`	`string`			Name of a secret containing a base64 encoded API Key to be included as value for the header controlled by `apiKeyHeaderName` (default: `X-API-Key`) on each request to the API
`codesPollInterval`	`go:`DurationString		`30s`	Interval between fetches of codes from API
`credentialsSecretName`	`string`			Name of a pre-existing Kubernetes Secret containing a username & password for authentication with upstream
`credentialsSecretPasswordKey`	`string`		`"username"`	If `credentialsSecretName` is specified: name of the item inside the Secret which holds the username
`credentialsSecretUsernameKey`	`string`		`"password"`	If `credentialsSecretName` is specified: name of the item inside the Secret which holds the password
`defaultGetTimeout`	`go:`DurationString		`60s`	Maximum duration of non-download `GET` request towards API (includes connect, header and body read)
`deltaPollInterval`	`go:`DurationString		`10s`	Interval between fetches of deltas from API
`errorRetryInterval`	`go:`DurationString		`1s`	Interval between retries when errors are encountered
`headers`	`dictionary`		`{}`	Set of `key:value` pairs to be set as headers on each request towards the API
`longGetTimeout`	`go:`DurationString		`40m`	Maximum duration of `GET` requests towards API, including those which fetch large files
`maxDeltaSize`	`integer`		`10000`	Maximum amount of deltas per fetch from API
`password`	`string`			Password to use for authentication
`url`	`string`	`yes`		URL of the upstream API
`username`	`string`			Username to use for authentication
`tlsconfig`	TLSConfig		`{}`	Upstream TLS configuration options

Upstream TLS Configuration

Parameters to configure TLS if the upstream has TLS enabled:

Parameter	Type	Default	Description
`ca`	`string`		CA in PEM format to use for validation. Use of `caSecretName` is recommended for security reasons
`caSecretName`	`string`		Name of a pre-existing Kubernetes Secret with a data item named `ca.crt` containing the CA in PEM format to use for validation
`insecure_skip_verify`	`boolean`	`false`	Skip validation of the storage backend's certificate chain and hostname