Skip to content

dstore-dist-eventforwarder

Configuring dstore-dist-eventforwarder to Send Events to Elasticsearch

The dstore-dist-eventforwarder is used to process the messages related to DNS filtering events for subscribers, and store them in Elasticsearch suitable for search/retrieval over a REST API provided by the dstore-dist-report-api component. If DNS filtering is not in use in your environment, then dstore-dist-eventforwarder and dstore-dist-report-api are not required to be configured.

dstore-dist-eventforwarder is configured using the configuration file eventforwarder.yml, which by default is located in /etc/pdns-dstore-dist/.

There are two types of message that can be sent to dstore-dist-eventforwarder:

  • Messages associated with a filtering event for a subscriber - These are used to store a record for each event in Elasticsearch, which can later be retrieved via the REST API of dstore-dist-report-api
  • All messages associated with subscribers - These are used to store a count of the total number of queries performed by each subscriber. Rather than send every one of these messages, it is more scalable to sample these messages in dstore-dist before sending to dstore-dist-event-forwarder. For example, sending only 1 in 50 messages. The exact sampling rate will vary per customer, based on total message volume. These messages are received using a different listen address than for filtering events.

To achieve the above, dstore-dist must be configured as follows:

  • A route must be created with a filter to send only response messages with tags and a requestorid (username); this ensures that messages which are not associated with a DNS filtering event are not sent to dstore-dist-eventforwarder.
  • A route must be configured to send to a destination with sampling enabled, to send all messages with a requestorid (username) to dstore-dist-eventforwarder using the address dedicated to query_counts.

An example configuration for dstore-dist is shown below: 

destinations:
  eventforwarder_filtered:
    addresses:
      - "192.168.1.2:8332"
  eventforwarder_sampled:
    sample: 50
    addresses:
      - "192.168.1.2:8334"
  routes:
    filtered:
      destinations:
        - eventforwarder_filtered
      filters:
        - has_tags: true
        - is_response: true
        - has_requestorid: true
    sampled:
      destinations:
        - eventforwarder_sampled
      filters:
        - is_response: true
        - has_requestorid: true

Configuring dstore-dist-eventforwarder consists of specifying the Elasticsearch/Opensearch connection details, and specifying how dstore-dist-eventforwarder will parse the tags present in the message to map to categories such as content filtering or malware. The sample rate used for the sampled messages should also be configured, and optionally the interval used to write query count data to Elasticsearch.

An example dstore-dist-eventforwarder configuration is shown below:

listen_addr: ":8332"

# Define filter types by tags.
filter_tags:
    contentfilter:
      - tagA
      - tagB
    malware: [tagC, tagD]
    blocklist:
      - tagE
    #phishing:
    #botnet:

elasticsearch:
    # Elasticsearch index where messages are sent.
    index: my-forwarder-index
    # Append the date to the index name to ensure a new index gets created each day
    add_date_to_index: true
    # The address of the elasticsearch server.
    addr: http://elasticsearch:9200
    # Auth username.
    user: elastic
    # Auth password.
    password: changeme

query_count:
    listen_addr: ":8334"
    sample_rate: 50
    query_interval: 6h

For more information on the configuration of dstore-dist-eventforwarder, see dstore-dist-eventforwarder Configuration.