egateway¶
egateway
allows you to search programmatically in the data stored by tcpdistro
. This can be done thanks to an http API.
/get-endpoint-data¶
Main entrypoint of the API. Search for DNS messages. By default, will look into customer traffic (queries and responses), which can be tuned with outgoing
and only_responses
parameters.
Parameters :
- q
search terms
- start
unix timestamp for the beginning of the timeframe you are looking for
- end
unix timestamp for the end of the timeframe you are looking for
- limit
limit the number of results
- preasons
search terms, separated by commas, within applied policies
- preasons_tags
search terms, separated by commas, within tags, which are not the applied policy
- has_preason
has been applied any policy
- only_responses
only search for responses (either incoming or direct)
- outgoing
only search for outgoing queries and incoming responses (recursive traffic)
Search terms is a list of keywords, separated by spaces with additional prefixes.
~address
search by answer address. example:~1.2.3.4
=message-id
search by message id. example:=123e4567-e89b-12d3-a456-426614174000
c:customer
search by requestor identifier. example:c:powerdns
d:device
search by device id. example:d:msisdn:07747012345
dn:device-name
search by device name field. example:dn:johns-computer
rcode:[!]code
search by dns response code. can search for all code except if!
is added. example:rcode:3
search for NXDomain responses,rcode:!0
search for all response code except NoError. See https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-6- the rest is considered to be either an IP address or a domain name. example:
1.2.3.4
,foo.example.com.
Examples:
$ http "http://127.0.0.1:8081/get-endpoint-data?q=%3D79d173da-9d83-4c89-ab6f-577a1a6eb59f"
HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri, 28 May 2021 09:57:42 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked
{
"items": [
{
"A": "2001:db8::1",
"B": "192.0.2.42",
"deviceId": "4005ffeeeeddaadddd",
"ecs": "2001:db8::1",
"from": "192.0.2.1",
"latencyUsec": -850884,
"msgType": 2,
"preason": "nxdomain",
"qid": "79d173da-9d83-4c89-ab6f-577a1a6eb59f",
"qr": true,
"queryTimeSec": 1605178842,
"queryTimeUsec": 850884,
"question": "23foobar.com",
"rcode": 1234,
"requestorId": "ncook",
"response": "0.0.0.0",
"serverId": "456",
"tags": [
"porn",
"gambling",
"OXP-platform-facebook"
],
"tcp": false,
"timeSec": 1605178842,
"timeUsec": 0,
"type": ""
}
],
"luaresult": {},
"msec": 0.306,
"number": 1,
"timegraph": []
}
$ http 'http://127.0.0.1:8081/get-endpoint-data?q=d:mac:000036000001%20rcode%3A!0'
HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri, 28 May 2021 10:06:17 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked
{
"items": [
{
"A": "2001:db8::1",
"B": "192.0.2.42",
"deviceId": "mac:000036000001",
"ecs": "2001:db8::1",
"from": "192.0.2.1",
"latencyUsec": -608116,
"msgType": 2,
"preason": "nxdomain",
"qid": "2e026b97-6b92-4be2-a7c4-97d6919b33ee",
"qr": true,
"queryTimeSec": 1605178887,
"queryTimeUsec": 608116,
"question": "29foobar.com",
"rcode": 1234,
"requestorId": "ncook",
"response": "0.0.0.0",
"serverId": "456",
"tags": [
"porn",
"gambling",
"OXP-platform-facebook"
],
"tcp": false,
"timeSec": 1605178887,
"timeUsec": 0,
"type": ""
},
[...]
],
"luaresult": {},
"msec": 0.366,
"number": 3,
"timegraph": []
}
/num-db¶
Some useful statistics on indexed entries and software version.
Example:
$ http 127.0.0.1:8081/num-db
HTTP/1.1 200 OK
Connection: keep-alive
Date: Thu, 27 May 2021 09:29:19 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked
{
"bytes": 5481,
"entries": 54,
"indexBytes": 0,
"version": "git"
}
/metrics¶
Prometheus metrics exporter.
Example:
$ http 127.0.0.1:8081/metrics
HTTP/1.1 200 OK
Connection: keep-alive
Date: Thu, 27 May 2021 09:25:07 GMT
Server: h2o/2.2.6
content-type: text/plain; charset=utf-8
transfer-encoding: chunked
# HELP dstore_egateway_queries_total Number of queries received
# TYPE dstore_egateway_queries_total counter
dstore_egateway_queries_total 0
# HELP dstore_egateway_queries_options_total Number of queries received by option
# TYPE dstore_egateway_queries_options_total counter
dstore_egateway_queries_options_total{option="timegraph"} 0
dstore_egateway_queries_options_total{option="lua"} 0
dstore_egateway_queries_options_total{option="response_only"} 0
# HELP dstore_egateway_queries_type_total Total number of searches by query type
# TYPE dstore_egateway_queries_type_total counter
dstore_egateway_queries_type_total{type="requestor_id"} 0
dstore_egateway_queries_type_total{type="device_id"} 0
dstore_egateway_queries_type_total{type="source"} 0
dstore_egateway_queries_type_total{type="domain"} 0
dstore_egateway_queries_type_total{type="qid"} 0
dstore_egateway_queries_type_total{type="content"} 0
dstore_egateway_queries_type_total{type="preason"} 0
# HELP dstore_egateway_records_matched_total Total number of records matched by queries
# TYPE dstore_egateway_records_matched_total counter
dstore_egateway_records_matched_total 0
# HELP dstore_egateway_records_returned_total Total number of records returned
# TYPE dstore_egateway_records_returned_total counter
dstore_egateway_records_returned_total 0
# HELP dstore_egateway_buckets_scanned_total Total number of buckets scanned
# TYPE dstore_egateway_buckets_scanned_total counter
dstore_egateway_buckets_scanned_total 0
# HELP dstore_egateway_bytes_scanned_total Total number of bytes scanned
# TYPE dstore_egateway_bytes_scanned_total counter
dstore_egateway_bytes_scanned_total 0
# HELP dstore_egateway_messages_scanned_total Total number of messages scanned
# TYPE dstore_egateway_messages_scanned_total counter
dstore_egateway_messages_scanned_total 0
# HELP dstore_egateway_queries_latency_seconds Histogram of queries latency.
# TYPE dstore_egateway_queries_latency_seconds histogram
dstore_egateway_queries_latency_seconds_bucket{le="0.01"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.03"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.05"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.1"} 0
dstore_egateway_queries_latency_seconds_bucket{le="0.3"} 0
dstore_egateway_queries_latency_seconds_bucket{le="1"} 0
dstore_egateway_queries_latency_seconds_bucket{le="3"} 0
dstore_egateway_queries_latency_seconds_bucket{le="5"} 0
dstore_egateway_queries_latency_seconds_bucket{le="10"} 0
dstore_egateway_queries_latency_seconds_bucket{le="20"} 0
dstore_egateway_queries_latency_seconds_bucket{le="30"} 0
dstore_egateway_queries_latency_seconds_bucket{le="60"} 0
dstore_egateway_queries_latency_seconds_bucket{le="120"} 0
dstore_egateway_queries_latency_seconds_bucket{le="180"} 0
dstore_egateway_queries_latency_seconds_bucket{le="+Inf"} 0
dstore_egateway_queries_latency_seconds_count 0
dstore_egateway_queries_latency_seconds_sum 0.000000
# HELP dstore_egateway_info Software version and configuration
# TYPE dstore_egateway_info gauge
dstore_egateway_info{version="git",hash_split="1024",storage_dir="./storage",question="1",message_id="1",requestor_id="1",policy="1",tags="1",store_queries="1"} 1
up 1
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1622107494
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 737280
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 0.003000
# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.006000
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
Authentication¶
If egateway
has been set up with an API key, you have to provide it through the x-api-key
http header.
Example:
$ http 127.0.0.1:8081/num-db
HTTP/1.1 403 Forbidden
Connection: keep-alive
Content-Length: 9
Date: Fri, 28 May 2021 10:35:26 GMT
Server: h2o/2.2.6
content-type: text/plain; charset=utf-8
forbidden
$ http -v 127.0.0.1:8081/num-db x-api-key:0xpassw0rd
GET /num-db HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: 127.0.0.1:8081
User-Agent: HTTPie/2.4.0
x-api-key: 0xpassw0rd
HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri, 28 May 2021 10:34:09 GMT
Server: h2o/2.2.6
content-type: application/json; charset=utf-8
transfer-encoding: chunked
{
"bytes": 5481,
"entries": 54,
"indexBytes": 0,
"version": "git"
}