Logging DNS messages with Protocol Buffers

The PowerDNS Recursor has the ability to emit a stream of protocol buffers messages over TCP, containing information about queries, answers and policy decisions.

Messages contain the IP address of the client initiating the query, the one on which the message was received, whether it was received over UDP or TCP, a timestamp and the qname, qtype and qclass of the question. In addition, messages related to responses contain the name, type, class and rdata of A, AAAA and CNAME records present in the response, as well as the response code.

Finally, if a RPZ or custom Lua policy has been applied, response messages also contain the applied policy name and some tags. This is particularly useful to detect and act on infected hosts.

Configuring Protocol Buffer logs

Protobuf export to a server is enabled using the protobufServer() directive:

protobufServer(server [, options]))

New in version 4.2.0.

Send protocol buffer messages to a server for incoming queries and/or outgoing responses. The client address may be masked using setProtobufMasks(), for anonymization purposes.

Parameters:
  • server (string) – The IP and port to connect to
  • options (table) – A table with key: value pairs with options.

Options:

  • timeout=2: int - Time in seconds to wait when sending a message
  • maxQueuedEntries=100: int - How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime=1: int - How long to wait, in seconds, between two reconnection attempts
  • taggedOnly=false: bool - Only entries with a policy or a policy tag set will be sent
  • asyncConnect: bool - When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  • logQueries=true: bool - Whether to export queries
  • logResponses=true: bool - Whether to export responses
  • exportTypes={'A', 'AAAA', 'CNAME'}: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
protobufServer(server[[[[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], maskV4=32], maskV6=128], asyncConnect=false], taggedOnly=false])

Deprecated since version 4.2.0.

Parameters:
  • server (string) – The IP and port to connect to
  • timeout (int) – Time in seconds to wait when sending a message
  • maxQueuedEntries (int) – How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime (int) – How long to wait, in seconds, between two reconnection attempts
  • maskV4 (int) – network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  • maskV6 (int) – Same as maskV4, but for IPv6. Defaults to 128.
  • taggedOnly (bool) – Only entries with a policy or a policy tag set will be sent.
  • asyncConnect (bool) – When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued..
setProtobufMasks(maskv4, maskV6)

New in version 4.2.0.

Parameters:
  • maskV4 (int) – network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  • maskV6 (int) – Same as maskV4, but for IPv6. Defaults to 128.

Logging outgoing queries and responses

While protobufServer() only exports the queries sent to the recursor from clients, with the corresponding responses, outgoingProtobufServer() can be used to export outgoing queries sent by the recursor to authoritative servers, along with the corresponding responses.

outgoingProtobufServer(server[, options])

New in version 4.2.0.

Send protocol buffer messages to a server for outgoing queries and/or incoming responses.

Parameters:
  • server (string) – The IP and port to connect to
  • options (table) – A table with key: value pairs with options.

Options:

  • timeout=2: int - Time in seconds to wait when sending a message
  • maxQueuedEntries=100: int - How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime=1: int - How long to wait, in seconds, between two reconnection attempts
  • taggedOnly=false: bool - Only entries with a policy or a policy tag set will be sent
  • asyncConnect: bool - When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  • logQueries=true: bool - Whether to export queries
  • logResponses=true: bool - Whether to export responses
  • exportTypes={'A', 'AAAA', 'CNAME'}: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
outgoingProtobufServer(server[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], asyncConnect=false])

Deprecated since version 4.2.0.

Parameters:
  • server (string) – The IP and port to connect to
  • timeout (int) – Time in seconds to wait when sending a message
  • maxQueuedEntries (int) – How many entries will be kept in memory if the server becomes unreachable
  • reconnectWaitTime (int) – How long to wait, in seconds, between two reconnection attempts
  • asyncConnect (bool) – When set to false (default) the first connection to the server during startup will block up to timeout seconds, otherwise the connection is done in a separate thread, after the first message has been queued..

Protobol Buffers Definition

The protocol buffers message types can be found in the dnsmessage.proto file and is included here:

/*
 * This file is part of PowerDNS or dnsdist.
 * Copyright -- PowerDNS.COM B.V. and its contributors
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of version 2 of the GNU General Public License as
 * published by the Free Software Foundation.
 *
 * In addition, for the avoidance of any doubt, permission is granted to
 * link this program with OpenSSL and to (re)distribute the binaries
 * produced as the result of such linking.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 */
syntax = "proto2";

message PBDNSMessage {
  enum Type {
    DNSQueryType = 1;
    DNSResponseType = 2;
    DNSOutgoingQueryType = 3;
    DNSIncomingResponseType = 4;
  }
  enum SocketFamily {
    INET = 1;                                   // IPv4 (RFC 791)
    INET6 = 2;                                  // IPv6 (RFC 2460)
  }
  enum SocketProtocol {
    UDP = 1;                                    // User Datagram Protocol (RFC 768)
    TCP = 2;                                    // Transmission Control Protocol (RFC 793)
  }
  enum PolicyType {
    UNKNOWN = 1;                                // No policy applied, or unknown type
    QNAME = 2;                                  // Policy matched on the QName
    CLIENTIP = 3;                               // Policy matched on the client IP
    RESPONSEIP = 4;                             // Policy matched on one of the IPs contained in the answer
    NSDNAME = 5;                                // Policy matched on the name of one nameserver involved
    NSIP = 6;                                   // Policy matched on the IP of one nameserver involved
  }
  required Type type = 1;
  optional bytes messageId = 2;                 // UUID, shared by the query and the response
  optional bytes serverIdentity = 3;            // ID of the server emitting the protobuf message
  optional SocketFamily socketFamily = 4;
  optional SocketProtocol socketProtocol = 5;
  optional bytes from = 6;                      // DNS requestor (client)
  optional bytes to = 7;                        // DNS responder (server)
  optional uint64 inBytes = 8;                  // Size of the query or response on the wire
  optional uint32 timeSec = 9;                  // Time of message reception (seconds since epoch)
  optional uint32 timeUsec = 10;                // Time of message reception (additional micro-seconds)
  optional uint32 id = 11;                      // ID of the query/response as found in the DNS header

  message DNSQuestion {
    optional string qName = 1;
    optional uint32 qType = 2;
    optional uint32 qClass = 3;
  }
  optional DNSQuestion question = 12;

  message DNSResponse {
    message DNSRR {
      optional string name = 1;
      optional uint32 type = 2;
      optional uint32 class = 3;
      optional uint32 ttl = 4;
      optional bytes rdata = 5;
      optional bool  udr = 6;                   // True if this is the first time this RR has been seen for this question
    }
    optional uint32 rcode = 1;
    repeated DNSRR rrs = 2;
    optional string appliedPolicy = 3;          // Filtering policy (RPZ or Lua) applied
    repeated string tags = 4;                   // Additional tags
    optional uint32 queryTimeSec = 5;           // Time of the corresponding query reception (seconds since epoch)
    optional uint32 queryTimeUsec = 6;          // Time of the corresponding query reception (additional micro-seconds)
    optional PolicyType appliedPolicyType = 7;  // Type of the filtering policy (RPZ or Lua) applied
  }

  optional DNSResponse response = 13;
  optional bytes originalRequestorSubnet = 14;  // EDNS Client Subnet value
  optional string requestorId = 15;             // Username of the requestor
  optional bytes initialRequestId = 16;         // UUID of the incoming query that initiated this outgoing query or incoming response
  optional bytes deviceId = 17;     		// Device ID of the requestor (could be mac address IP address or e.g. IMEI)
  optional bool  newlyObservedDomain = 18;      // True if the domain has not been seen before
}