egateway

egateway allows you to search programmatically in the data stored by tcpdistro. This can be done thanks to an http API.

/get-endpoint-data

Main entrypoint of the API. Search for DNS messages. By default, will look into customer traffic (queries and responses), which can be tuned with outgoing and only_responses parameters.

Parameters : - q search terms - start unix timestamp for the beginning of the timeframe you are looking for - end unix timestamp for the end of the timeframe you are looking for - limit limit the number of results - preasons search terms, separated by commas, within applied policies - preasons_tags search terms, separated by commas, within tags, which are not the applied policy - has_preason has been applied any policy - only_responses only search for responses (either incoming or direct) - outgoing only search for outgoing queries and incoming responses (recursive traffic)

Search terms is a list of keywords, separated by spaces with additional prefixes.

  • ~address search by answer address. example: ~1.2.3.4
  • =message-id search by message id. example: =123e4567-e89b-12d3-a456-426614174000
  • c:customer search by requestor identifier. example: c:powerdns
  • d:device search by device id. example: d:msisdn:07747012345
  • dn:device-name search by device name field. example: dn:johns-computer
  • rcode:[!]code search by dns response code. can search for all code except if ! is added. example: rcode:3 search for NXDomain responses, rcode:!0 search for all response code except NoError. See https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-6
  • the rest is considered to be either an IP address or a domain name. example: 1.2.3.4, foo.example.com.

Examples:

$ http "http://127.0.0.1:8081/get-endpoint-data?q=%3D79d173da-9d83-4c89-ab6f-577a1a6eb59f"
HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri, 28 May 2021 09:57:42 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked

{
  "items": [
    {
      "A": "2001:db8::1",
      "B": "192.0.2.42",
      "deviceId": "4005ffeeeeddaadddd",
      "ecs": "2001:db8::1",
      "from": "192.0.2.1",
      "latencyUsec": -850884,
      "msgType": 2,
      "preason": "nxdomain",
      "qid": "79d173da-9d83-4c89-ab6f-577a1a6eb59f",
      "qr": true,
      "queryTimeSec": 1605178842,
      "queryTimeUsec": 850884,
      "question": "23foobar.com",
      "rcode": 1234,
      "requestorId": "ncook",
      "response": "0.0.0.0",
      "serverId": "456",
      "tags": [
        "porn",
        "gambling",
        "OXP-platform-facebook"
      ],
      "tcp": false,
      "timeSec": 1605178842,
      "timeUsec": 0,
      "type": ""
    }
  ],
  "luaresult": {},
  "msec": 0.306,
  "number": 1,
  "timegraph": []
}

$ http 'http://127.0.0.1:8081/get-endpoint-data?q=d:mac:000036000001%20rcode%3A!0'
HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri, 28 May 2021 10:06:17 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked

{
  "items": [
    {
      "A": "2001:db8::1",
      "B": "192.0.2.42",
      "deviceId": "mac:000036000001",
      "ecs": "2001:db8::1",
      "from": "192.0.2.1",
      "latencyUsec": -608116,
      "msgType": 2,
      "preason": "nxdomain",
      "qid": "2e026b97-6b92-4be2-a7c4-97d6919b33ee",
      "qr": true,
      "queryTimeSec": 1605178887,
      "queryTimeUsec": 608116,
      "question": "29foobar.com",
      "rcode": 1234,
      "requestorId": "ncook",
      "response": "0.0.0.0",
      "serverId": "456",
      "tags": [
        "porn",
        "gambling",
        "OXP-platform-facebook"
      ],
      "tcp": false,
      "timeSec": 1605178887,
      "timeUsec": 0,
      "type": ""
    },
    [...]
  ],
  "luaresult": {},
  "msec": 0.366,
  "number": 3,
  "timegraph": []
}

/num-db

Some useful statistics on indexed entries and software version.

Example:

$ http 127.0.0.1:8081/num-db
HTTP/1.1 200 OK
Connection: keep-alive
Date: Thu, 27 May 2021 09:29:19 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked

{
  "bytes": 5481,
  "entries": 54,
  "indexBytes": 0,
  "version": "git"
}

/metrics

Prometheus metrics exporter.

Example:

$ http 127.0.0.1:8081/metrics
HTTP/1.1 200 OK
Connection: keep-alive
Date: Thu, 27 May 2021 09:25:07 GMT
Server: h2o/2.2.6
content-type: text/plain; charset=utf-8
transfer-encoding: chunked

# HELP dstore_egateway_queries_total Number of queries received
# TYPE dstore_egateway_queries_total counter
dstore_egateway_queries_total 0
# HELP dstore_egateway_queries_options_total Number of queries received by option
# TYPE dstore_egateway_queries_options_total counter
dstore_egateway_queries_options_total{option="timegraph"} 0
dstore_egateway_queries_options_total{option="lua"} 0
dstore_egateway_queries_options_total{option="response_only"} 0
# HELP dstore_egateway_queries_type_total Total number of searches by query type
# TYPE dstore_egateway_queries_type_total counter
dstore_egateway_queries_type_total{type="requestor_id"} 0
dstore_egateway_queries_type_total{type="device_id"} 0
dstore_egateway_queries_type_total{type="source"} 0
dstore_egateway_queries_type_total{type="domain"} 0
dstore_egateway_queries_type_total{type="qid"} 0
dstore_egateway_queries_type_total{type="content"} 0
dstore_egateway_queries_type_total{type="preason"} 0
# HELP dstore_egateway_records_matched_total Total number of records matched by queries
# TYPE dstore_egateway_records_matched_total counter
dstore_egateway_records_matched_total 0
# HELP dstore_egateway_records_returned_total Total number of records returned
# TYPE dstore_egateway_records_returned_total counter
dstore_egateway_records_returned_total 0
# HELP dstore_egateway_buckets_scanned_total Total number of buckets scanned
# TYPE dstore_egateway_buckets_scanned_total counter
dstore_egateway_buckets_scanned_total 0
# HELP dstore_egateway_bytes_scanned_total Total number of bytes scanned
# TYPE dstore_egateway_bytes_scanned_total counter
dstore_egateway_bytes_scanned_total 0
# HELP dstore_egateway_messages_scanned_total Total number of messages scanned
# TYPE dstore_egateway_messages_scanned_total counter
dstore_egateway_messages_scanned_total 0
# HELP dstore_egateway_queries_latency_seconds Histogram of queries latency.
# TYPE dstore_egateway_queries_latency_seconds histogram
dstore_egateway_queries_latency_seconds_bucket{le="0.01"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.03"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.05"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.1"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.3"} 0
dstore_egateway_queries_latency_seconds_bucket{le="1"} 0
dstore_egateway_queries_latency_seconds_bucket{le="3"} 0
dstore_egateway_queries_latency_seconds_bucket{le="5"} 0
dstore_egateway_queries_latency_seconds_bucket{le="10"} 0
dstore_egateway_queries_latency_seconds_bucket{le="20"} 0
dstore_egateway_queries_latency_seconds_bucket{le="30"} 0
dstore_egateway_queries_latency_seconds_bucket{le="60"} 0
dstore_egateway_queries_latency_seconds_bucket{le="120"} 0
dstore_egateway_queries_latency_seconds_bucket{le="180"} 0
dstore_egateway_queries_latency_seconds_bucket{le="+Inf"} 0
dstore_egateway_queries_latency_seconds_count 0
dstore_egateway_queries_latency_seconds_sum 0.000000
# HELP dstore_egateway_info Software version and configuration
# TYPE dstore_egateway_info gauge
dstore_egateway_info{version="git",hash_split="1024",storage_dir="./storage",question="1",message_id="1",requestor_id="1",policy="1",tags="1",store_queries="1"} 1
up 1
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1622107494
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 737280
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 0.003000
# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.006000
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9

Authentication

If egateway has been set up with an API key, you have to provide it through the x-api-key http header.

Example:

$ http 127.0.0.1:8081/num-db
HTTP/1.1 403 Forbidden
Connection: keep-alive
Content-Length: 9
Date: Fri, 28 May 2021 10:35:26 GMT
Server: h2o/2.2.6
content-type: text/plain; charset=utf-8

forbidden

$ http -v 127.0.0.1:8081/num-db x-api-key:0xpassw0rd
GET /num-db HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: 127.0.0.1:8081
User-Agent: HTTPie/2.4.0
x-api-key: 0xpassw0rd



HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri, 28 May 2021 10:34:09 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked

{
  "bytes": 5481,
  "entries": 54,
  "indexBytes": 0,
  "version": "git"
}