How it works
============

:program:`dstore` accepts *events* over the network and subsequently stores them.
Each event contains a DNS message (source address, query name, query type, etc.) augmented by performance counters all of which is committed to storage.

:program:`dstore` receives events wrapped in `Protocol Buffers <https://en.m.wikipedia.org/wiki/Protocol_Buffersprotobuf>`__ (a.k.a. protobuf).
These are a language-neutral, platform-neutral, extensible mechanism for serializing structured data.

The events destined for :program:`dstore` can be emitted by a variety of sources:

-  PowerDNS Recursor
-  PowerDNS dnsdist
-  PowerDNS Authoritative Server
-  :program:`dnspcap2protobuf` from the PowerDNS distribution

| |Architecture|

Components
----------

:program:`dstore` consists of a number of components which interact with each other.
Depending on the functionality desired, some of these components are optional.
For example, if you are interested in storage of DNS messages and retrieval via the command-line only, you run :program:`tcpdistro` and :program:`dgrep`, omitting the UI server and :program:`egateway`.

We next describe the individual components which make up :program:`dstore`.

``tcpdistro``
~~~~~~~~~~~~~

This is the primary daemon which listens for events on TCP port 2000.
It is responsible for listening for events, decoding protobufs, and managing the local storage of events in the data store.

Tightly coupled to :program:`tcpdistro` is the :program:`dstore` utility which is used by system administrators to clean up (or rotate) the storage facility.

``dgrep``
~~~~~~~~~

You use the :program:`dgrep` utility to query, from the command line, the :program:`dstore` storage facility.
It’s a simple, but very fast tool, which emits JSON objects containing responses of previously stored DNS messages.

.. code:: json

   {
     "items": [
       {
         "answer": "",
         "origRequestor": "192.0.2.213",
         "preason": "",
         "qid": "278387a5-3080-4111-b8e0-1a25c1fb2340",
         "qr": false,
         "question": "www.powerdns.com.",
         "rcode": 0,
         "requestor": "192.168.1.130",
         "responder": "192.168.1.206",
         "tags": [],
         "timeSec": 1466489029,
         "timeUsec": 712294,
         "type": "AAAA"
       }
     ],
     "luaresult": {},
     "msec": 1.111,
     "number": 1
   }

``dcat``
~~~~~~~~

You can use the :program:`dcat` utility to concatenate dstore files and print them on the standard output.

Examples :

To read the raw dstore file and print its content as JSON :

.. code:: bash

   ./src/dcat --quiet <path-to-raw-dstore-file> | jq


You can also print them as raw protobuf messages using the `--raw` switch :

.. code:: bash

   ./src/dcat --raw --quiet <path-to-raw-dstore-file> | ./go/src/dnspb2json/dnspb2json | jq

``egateway``
~~~~~~~~~~~~

The optional :program:`egateway` is a HTTP gateway to the DNS messages in :program:`dstore`.
It is intended to run on the machine with the storage, and it listens on a configurable TCP port, which defaults to 8081.

UI server (dstore-web)
~~~~~~~~~~~~~~~~~~~~~~

:program:`dstore-web` is a Python Django-based application which serves as a Web back-end for a Web browser on the one hand, and talks to :program:`egateway` on the other.
The UI server can run on a separate machine from the egateway, providing it can communicate with :program:`egateway`.

| |dstore Web UI|

The result provides an attractive and fast Web-based user-interface to :program:`dstore`.

dnspcap2protobuf
~~~~~~~~~~~~~~~~

The :program:`dnspcap2protobuf` utility from the PowerDNS distribution reads a pcap file and converts it to *protobuf* format which you can subsequently feed to :program:`dstore`:

.. code:: bash

   tcpdump -i enp0s25 udp port 53 -w example.pcap
   dnspcap2protobuf example.pcap /dev/stdout | nc 127.0.0.1 2000


.. |Architecture| image:: assets/dstore-architecture.png
   :width: 80.0%
.. |dstore Web UI| image:: assets/web-ui.png