Add DNS resolution poller

[mikehale]

Ref https://status.heroku.com/incidents/698.

We used to have an snmp based host -W 1 google.com > /dev/null nagios check on all instances that reported success or failure of attempting to resolve DNS on each host. Trusty snmpd does not seem to have support for these types of checks anymore and we never alerted on these checks anyway so shh seems like a good candidate for this type of boolean check.

[cmeckhardt]

This comes out of incident 698.

[freeformz]

Is the purpose of this dns pollerto ascertain if a known host can be looked up using the local resolver and it's configured DNS servers?

Or should the poller handle the DNS itself and be provided the hostname to lookup and the DNS servers that are relevant?

What should it report? Query time? number of responses? something else?

[mikehale]

The purpose is to ascertain if the local DNS system is functioning properly. Specifically to have a good idea if calls by application code to getaddrinfo are likely to succeed or fail within a reasonable amount of time.

Perhaps reporting response time with a somewhat high timeout of 10s would be a good start.