OX Feed Mirror

OX Feed Mirror allows you to mirror feeds provided by PowerDNS/OX or feeds managed by other feed generators such as Command Source.

The main functionality is captured in the below diagram:

Image title — Overview of OX Feed Mirror

In the diagram you can see the main 2 components:

oxfeed-mirror container is responsible for monitoring the configured feeds and synchronising them to the local "feed storage"
webserver container is responsible for exposing the feeds stored in "feed storage"

When deployed, OX Feed Mirror will expose a feed URL at the following path for each configured upstream feed:

[Service / Ingress]/feed

For example, see the following configuration:

oxfeedMirrors:
  mymirror:
    feeds:
      - name: examplefeed
        url: "https://my.feeds.endpoint/feed/test"
        credentialsSecretName: my-oxfeed-secret
      - name: extrafeed
        url: "https://another.feeds.endpoint/feed/feedtwo"
        credentialsSecretName: extra-oxfeed-secret
        pollInterval: 3m

This will expose 2 feeds:

[Service / Ingress]/feed/examplefeed
[Service / Ingress]/feed/extrafeed

Services created for OX Feed Mirror are as follows:

HTTP: Port 8080 on Service oxfeedmirror-[Instance Set Name]
HTTPS (if enabled): Port 8443 on Service oxfeedmirror-[Instance Set Name]-https

Configuration Reference

Instance Sets

Instances of OX Feed Mirror can be defined under the root node oxfeedMirrors. Each instance set should be a key-value pair, with:

key: Name of the instance set
value: Dictionary holding the configuration of the instance set

oxfeedMirrors:
  mymirror:
    < OX Feed Mirror configuration >

In a minimal configuration you will need to specify at least the url of a feed to mirror and the accompanying credentials (preferably via a Secret)

For example:

oxfeedMirrors:
  mymirror:
    feeds:
      - name: examplefeed
        url: "https://my.feeds.endpoint/feed/test"
        credentialsSecretName: my-oxfeed-secret

In the above example OX Feed Mirror will authenticate to the OX Feed at https://my.feeds.endpoint/feed/test using the username and password stored in a Kubernetes Secret named my-oxfeeds-secret. After an initial full sync, it will constantly monitor the feed and synchronize the local copy accordingly.

Since feeds is a list, you can define multiple feeds to be mirrored by the same OX Feed Mirror instance set.

Parameters which can be used to further configure OX Feed Mirror instances for a specific instance set are shown in the below table.

Parameter	Type	Default	Description
`affinity`	`k8s:`Affinity		pod affinity (Kubernetes docs: Affinity and anti-affinity). If unset, a default anti-affinity is applied using `antiAffinityPreset` to spread pods across nodes
`agentLogLevel`	`string`	`"info"`	Verbosity of logging for the agent container. Available options: `"debug"` `"info"` `"warn"` `"error"`
`agentLogFormat`	`string`	`"text"`	Format of logging for the agent container. Available options: `"text"` `"json"`
`agentResources`	`k8s:`Resources	`limits: cpu: 250m memory: 256Mi`	Resources allocated to the agent container if `resourceDefaults` (global) is `true`
`antiAffinityPreset`	`string`	`"preferred"`	pod anti affinity preset. Available options: `"preferred"` `"required"`
`containerSecurityContext`	`k8s:`SecurityContext	`allowPrivilegeEscalation: false readOnlyRootFilesystem: true seccompProfile: type: RuntimeDefault capabilities: drop: - "ALL"`	SecurityContext applied to each container
`defaultPollInterval`	`go:`DurationString	`60s`	Interval between checks against each configured feeds for updates
`disableSticky`	`boolean`	`false`	If `true`, disable the operator-controlled selection of a primary pod to handle all inbound traffic
`feeds`	List of Feed	`[]`	Upstream feed(s) configuration
`hostNetwork`	`boolean`	`false`	Use host networking for pods
`ingress`	`k8s:`IngressSpec	`{}`	Ingress configuration
`logFormat`	`string`	`"human"`	Format of logging. Available options: `"human"` `"json"`
`logLevel`	`string`	`"info"`	Level of logging. Available options: `"debug"` `"info"` `"warn"` `"error"`
`nodeSelector`	`k8s:`NodeSelector	`{}`	Kubernetes pod nodeSelector
`podAnnotations`	`k8s:`Annotations	`{}`	Annotations to be added to each pod
`podDisruptionBudget`	`k8s:`PodDisruptionBudgetSpec	`{}`	Spec of PodDisruptionBudget to be applied to deployment
`podLabels`	`k8s:`Labels	`{}`	Labels to be added to each pod
`podSecurityContext`	`k8s:`PodSecurityContext	`fsGroup: 953 runAsUser: 953 runAsGroup: 953 runAsNonRoot: true`	SecurityContext applied to each pod
`readyInterval`	`integer`	`5`	How often readiness of the OX Feed Mirror should be calculated in seconds
`replicas`	`integer`	`2`	Default number of replicas in a Deployment
`resources`	`k8s:`Resources	`limits: cpu: 1 memory: 2Gi`	Resources allocated to the oxfeed-mirror container if `resourceDefaults` (global) is `true`
`service`	Service	`type: ClusterIP`	Service configuration
`serviceHTTPS`	Service	`type: ClusterIP`	HTTPS Service configuration Note: Only created when `tls.enabled = true`
`serviceLabels`	`k8s:`Labels	`{}`	Labels to be added to each service
`tls`	Inbound TLS	`enabled: false`	TLS configuration for inbound HTTPS traffic
`tolerations`	List of `k8s:`Tolerations	`[]`	Kubernetes pod Tolerations
`topologySpreadConstraints`	List of `k8s:`TopologySpreadConstraint	`[]`	Kubernetes pod topology spread constraints
`unreadyAfterNoSuccessPeriod`	`integer`	`0`	If OX Feed Mirror has not successfully ran for this many seconds, mark the pod as Unready. Disabled if 0 (default)
`users`	List of `string`	`["feeduser"]`	List of usernames for which a Secret containing the username and a generated password is created, which is given access to the feed. Note: The generated passwords are stored in base64 encoded form in the secret. Make sure you decode them
`userSecrets`	List of `string`	`[]`	List of names of Secrets containing `username` and `password` items which should act as credentials allowed to access the feed
`webserverHTTPExtra`	List of `string`	`[]`	Extra lines of NGINX configuration to be injected into the `server` directive for the HTTP traffic. Should not include a trailing `;`. For example: `webserverHTTPExtra: - "add_header X-HTTP-1 shortval" - "add_header X-HTTP-2 'longer value'"`
`webserverHTTPSExtra`	List of `string`	`[]`	Extra lines of NGINX configuration to be injected into the `server` directive for the HTTPS traffic. Should not include a trailing `;`. For example: `webserverHTTPExtra: - "add_header X-HTTPS-1 shortval" - "add_header X-HTTPS-2 'longer value'"`
`webserverResources`	`k8s:`Resources	`limits: cpu: 250m memory: 256Mi`	Resources allocated to the webserver container if `resourceDefaults` (global) is `true`
`webserverSecretName`	`string`		Name of a secret containing `username: base64_password` pairs to be granted access to the feed. `base64_password` should be the base64 encoded representation of the actual password Deprecated: Replaced by `userSecrets`
`webserverUsers`	List of `string`	`["csuser"]`	List of usernames for which a password should be generated and allowed access to the feed. By default a user named `csuser` will be created with a randomized password which can be obtained from the corresponding secret. Note: The generated passwords are stored in base64 encoded form in the secret. Make sure you decode them Deprecated: Replaced by `users`

Feed Configuration

Parameters to configure the upstream feeds to be mirrored. For example:

oxfeedMirrors:
  mymirror:
    feeds:
      - name: examplefeed
        url: "https://my.feeds.endpoint/feed/test"
        credentialsSecretName: my-oxfeed-secret
      - name: extrafeed
        url: "https://another.feeds.endpoint/feed/feedtwo"
        credentialsSecretName: extra-oxfeed-secret
        pollInterval: 3m

Parameter	Type	Required	Default	Description
`credentialsSecretName`	`string`	`yes`		Name of a pre-existing Kubernetes Secret containing a username & password for authentication with feed
`credentialsSecretPasswordKey`	`string`		`"username"`	If `credentialsSecretName` is specified: name of the item inside the Secret which holds the username
`credentialsSecretUsernameKey`	`string`		`"password"`	If `credentialsSecretName` is specified: name of the item inside the Secret which holds the password
`expiry`	Feed Expiry		`{}`	Expiry configuration
`fetchTimeout`	`go:`DurationString		`4h`	Maximum duration of an attempt to fetch the upstream feed. Defaults to a conservatively high value to allow for a large feed combined with a slow connection
`password`	`string`			Password to use for authentication
`pollInterval`	`go:`DurationString		`60s`	Interval between checks against upstream feed for updates (has precedence over `defaultPollInterval`)
`tlsconfig`	TLSConfig		`{}`	Upstream feed TLS configuration options
`url`	`string`	`yes`		URL of upstream feed
`username`	`string`			Username to use for authentication

Feed Expiry Configuration

Parameters to configure the expiration of feeds. For example:

oxfeedMirrors:
  mymirror:
    feeds:
      - name: examplefeed
        url: "https://my.feeds.endpoint/feed/test"
        credentialsSecretName: my-oxfeed-secret
        expiry:
          snapshots:
            must_keep_count: 10
            max_keep_count: 20
          deltas:
            must_keep_count: 40
            max_keep_count: 100

The expiry node allows for configuration of expiration of snapshots and deltas, corresponding to the full snapshots & accompanying incremental deltas offered by the mirror. By default, the expiration mechanism is disabled.

Both snapshots and deltas can be configured with the following parameters:

Parameter	Type	Description
`max_keep_count`	`integer`	Remove the oldest updates if this value is configured and the current count exceeds this value. `must_` parameters supersede this.
`max_keep_interval`	`go:`DurationString	Remove updates if this value is configured and there are updates present which are older. `must_` parameters supersede this.
`must_keep_count`	`integer`	Minimum number of updates to retain
`must_keep_interval`	`go:`DurationString	Minimum age of an update before it can be considered for expiration

Feed TLS Configuration

Parameters to configure TLS if the upstream feed has TLS enabled:

Parameter	Type	Default	Description
`ca`	`string`		CA in PEM format to use for validation. Use of `caSecretName` is recommended for security reasons
`caSecretName`	`string`		Name of a pre-existing Kubernetes Secret with a data item named `ca.crt` containing the CA in PEM format to use for validation
`insecure_skip_verify`	`boolean`	`false`	Skip validation of the storage backend's certificate chain and hostname

Inbound TLS

Parameters to configure TLS for inbound traffic. When enabled is set to true and a certificate source is configured, an additional Service with the name suffix -https will be created. An example:

oxfeedMirrors:
  mymirror:
    feeds:
      - name: examplefeed
        url: "https://my.feeds.endpoint/feed/test"
        credentialsSecretName: my-oxfeed-secret

    tls:
      enabled: true
      certSecretName: my-httpapi-certificate

In the above example the certificate present in Secret my-httpapi-certificate will be attempted to be used to start a TLS-enabled listener.

Parameter	Type	Default	Description
`certSecretName`	`string`		Name of a Secret object containing a certificate (must contain the `tls.key`, `tls.crt` items)
`certManager`	`boolean`	`false`	Toggle to have a request created for Certmanager to provision a certificate. By default, this will request for a Certificate covering the following: - `httpapi-[Name of instance set]-https` - `httpapi-[Name of instance set]-https.[Namespace]` - `httpapi-[Name of instance set]-https.[Namespace].svc` Additional entries can be configured using `extraDNSNames`
`enabled`	`boolean`	`false`	Toggle to enable TLS If set to `true`, a `certSecretName` must be set or `certManager` must be set to `true` to ensure a valid certificate is available
`extraDNSNames`	List of `string`	`[]`	List of additional entries to be added to the Certificate requested from Certmanager
`issuerGroup`	`string`	`"cert-manager.io"`	Group to which issuer specified under `issuerKind` belongs Default value is inherited from the global `certManager` configuration
`issuerKind`	`string`	`"ClusterIssuer"`	Type of Certmanager issuer to request a Certificate from Default value is inherited from the global `certManager` configuration
`issuerName`	`string`	`""`	Name of the issuer from which to request a Certificate Default value is inherited from the global `certManager` configuration
`certSpecExtra`	CertificateSpec	`{}`	Extra configuration to be injected into the Certmanager Certificate object's `spec` field. Disallowed options: `"secretName"` `"commonName"` `"dnsNames"` `"issuerRef"` (These are configured automatically and/or via other options)
`certLabels`	`k8s:`Labels	`{}`	Extra labels for the Certmanager Certificate object
`certAnnotations`	`k8s:`Annotations	`{}`	Extra annotations for the Certmanager Certificate object

Service Configuration

Parameters to configure the service object for this deployment. For example:

oxfeedMirrors:
  mymirror:
    feeds:
      - name: examplefeed
        url: "https://my.feeds.endpoint/feed/test"
        credentialsSecretName: my-oxfeed-secret
    service:
      type: LoadBalancer
      annotations:
        metallb.universe.tf/address-pool: name_of_pool

Parameter	Type	Default	Description
`allocateLoadBalancerNodePorts`	`boolean`	`true`	If true, services with type `LoadBalancer` automatically assign NodePorts. Can be set to `false` if the LoadBalancer provider does not rely on NodePorts
`annotations`	`k8s:`Annotations	`{}`	Annotations for the service
`clusterIP`	`string`		Static cluster IP, must be in the cluster's range of cluster IPs and not in use. Randomly assigned when not specified.
`clusterIPs`	List of `string`		List of static cluster IPs, must be in the cluster's range of cluster IPs and not in use.
`externalIPs`	List of `string`		List of IP addresses for which nodes in the cluster will also accept traffic for this service. These IPs are not managed by Kubernetes and must be user-defined on the cluster's nodes
`externalTrafficPolicy`	`string`	`Cluster`	Can be set to `Local` to let nodes distribute traffic received on one of the externally-facing addresses (`NodePort` and `LoadBalancer`) solely to endpoints on the node itself
`healthCheckNodePort`	`integer`		For services with type `LoadBalancer` and externalTrafficPolicy `Local` you can configure this value to choose a static port for the NodePort which external systems (LoadBalancer provider mainly) can use to determine which node holds endpoints for this service
`internalTrafficPolicy`	`string`	`Cluster`	Can be set to `Local` to let nodes distribute traffic received on the `ClusterIP` solely to endpoints on the node itself
`ipv4`	`boolean`	`false`	If true, force the Service to include support for IPv4, ignoring globally configured IP Family settings and/or cluster defaults. If `ipv4` is set to `true` and `ipv6` remains `false`, the result will be an `ipv4`-only SingleStack Service. If both are `false`, global settings and/or cluster defaults are used. If both are `true`, a PreferDualStack Service is created
`ipv6`	`boolean`	`false`	If true, force the Service to include support for IPv6, ignoring globally configured IP Family settings and/or cluster defaults. If `ipv6` is set to `true` and `ipv4` remains `false`, the result will be an `ipv6`-only SingleStack Service. If both are `false`, global settings and/or cluster defaults are used. If both are `true`, a PreferDualStack Service is created
`labels`	`k8s:`Labels	`{}`	Labels to be added to the service
`loadBalancerIP`	`string`		Deprecated Kubernetes feature, available for backwards compatibility: IP address to attempt to claim for use by this LoadBalancer. Replaced by `annotations` specific to each LoadBalancer provider
`loadBalancerSourceRanges`	List of `string`		If supported by the LoadBalancer provider, restrict traffic to this LoadBalancer to these ranges
`loadBalancerClass`	`string`		Used to select a non-default type of LoadBalancer class to ensure the appropriate LoadBalancer provisioner attempt to manage this LoadBalancer service
`publishNotReadyAddresses`	`boolean`	`false`	Service is populated with endpoints regardless of readiness state
`sessionAffinity`	`string`	`None`	Can be set to `ClientIP` to attempt to maintain session affinity.
`sessionAffinityConfig`	`k8s:`SessionAffinityConfig	`{}`	Configuration of session affinity
`type`	`string`	`ClusterIP`	Type of service. Available options: `"ClusterIP"` `"LoadBalancer"` `"NodePort"`