A functional and useful dashboard for pfSense that utilizes influxdb, grafana and telegraf
Grafana 7.4.3 Influxdb 1.8.3
This is just a summary, for more details look at the commits.
Due to the update in the Gateway plugin (move from py to php), you may need to drop your gateways measurement.
I renamed many of the columns to reflect what's being logged by pfBlockerNG-devel and fixed some parsing bugs that cause lines to be skipped due to inconsistent log formatting. As a result, the measurements ipblocklog and dnsbllog have been replaced with tailipblocklog and taildnsbllog respectively.
I dropped the old measurements: ipblocklog, dnsbl_log
If you cannot live without this data, you could use the panels from this commit and not update the config. Read my note about the Logparser Input Plugin above!
If you want to load the complete logs files, you could probably change the telegraf config to:
from_beginning = false
from_beginning = true
I'm sure you can even rename the measurements, columns and update the tags, but that's beyond my influx capabilities.
Kubernetes deployed locally with these instructions
Stripped yaml templates used to deploy my homelab (including Influx and Grafana) are here
grafana-pfSense: image: "grafana/grafana:7.4.3" container_name: grafana hostname: grafana mem_limit: 4gb ports: - "3000:3000" environment: TZ: "America/New_York" GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource,grafana-piechart-panel,grafana-worldmap-panel" GF_PATHS_DATA: "/var/lib/grafana" GF_DEFAULT_INSTANCE_NAME: "home" GF_ANALYTICS_REPORTING_ENABLED: "false" GF_SERVER_ENABLE_GZIP: "true" GF_SERVER_DOMAIN: "home.mydomain" volumes: - '/share/ContainerData/grafana:/var/lib/grafana' logging: driver: "json-file" options: max-size: "100M" network_mode: bridge
influxdb-pfsense: image: "influxdb:1.8.3-alpine" container_name: influxdb hostname: influxdb mem_limit: 10gb ports: - "2003:2003" - "8086:8086" environment: TZ: "America/New_York" INFLUXDB_DATA_QUERY_LOG_ENABLED: "false" INFLUXDB_REPORTING_DISABLED: "true" INFLUXDB_HTTP_AUTH_ENABLED: "true" INFLUXDB_ADMIN_USER: "admin" INFLUXDB_ADMIN_PASSWORD: "adminpassword" INFLUXDB_USER: "pfsense" INFLUXDB_USER_PASSWORD: "pfsenseuserpassword" INFLUXDB_DB: "pfsense" volumes: - '/share/ContainerData/influxdb:/var/lib/influxdb' logging: driver: "json-file" options: max-size: "100M" network_mode: bridge
Make sure you are using pfBlockerNG-devel
The Config for the dashboard relies on the variables defined within the dashboard in Grafana. When importing the dashboard, make sure to select your datasource.
Dashboard Settings -> Variables
WAN - $WAN is a static variable defined so that a separate dashboard panel can be created for WAN interfaces stats. Use a comma-separated list for multiple WAN interfaces.
LANInterfaces - $LANInterfaces uses a regex to remove any interfaces you don't want to be grouped as LAN. The filtering happens in the "Regex" field. I use a negative lookahead regex to match the interfaces I want excluded. It should be pretty easy to understand what you need to do here. I have excluded igb0 (WAN) and igb1,igb2,igb3 (only used to host vlans).
After writing this up, I realize I need to change this variable name, it's just not going to happen right now.
In the /config directory you will find all of the additional telegraf config. In pfSense, under Services -> Telegraf, at the bottom of the page with the teeny tiny text box is where you paste in the included config.
I also included the config for Unbound DNS and it's commented out. I'm not currently using it, but it's fully functional, just uncomment if you want to use it.
Plugins get copied to your pfSense system
I put all my plugins in /usr/local/bin and set them to 555
I also included a wrapper script for Unbound DNS. I'm not currently using it, but it's fully functional.
# cat -e /usr/local/bin/telegraf_pfinterface.php
If you get no good output from running the plugin directly, try the following command before moving to the below step.
# telegraf --test --config /usr/local/etc/telegraf.conf
To troubleshoot plugins further, add the following lines to the agent block in /usr/local/etc/telegraf.conf and send a HUP to the telegraf pid. You're going to need to do this from a ssh shell. One you update the config you are going to need to tell telegraf to read the new configs. If you restart telegraf from pfSense, this will not work since it will overwrite your changes.
debug = true quiet = false logfile = "/var/log/telegraf/telegraf.log"
# ps aux | grep '[t]elegraf.conf' # kill -HUP
Now go read /var/log/telegraf/telegraf.log
When in doubt, run a few queries to see if the data you are looking for is being populated.
bash-4.4# influx Connected to http://localhost:8086 version 1.8.3 InfluxDB shell version: 1.8.3 > auth username: admin password: > show databases name: databases name ---- pfsense _internal > use pfsense Using database pfsense > show measurements name: measurements name ---- cpu disk diskio gateways interface mem net netstat pf processes swap system tail_dnsbl_log tail_ip_block_log temperature > select * from system limit 20 name: system time host load1 load15 load5 n_cpus n_users uptime uptime_format ---- ---- ----- ------ ----- ------ ------- ------ ------------- 1585272640000000000 pfSense.home 0.0615234375 0.07861328125 0.0791015625 4 1 196870 2 days, 6:41 1585272650000000000 pfSense.home 0.05126953125 0.07763671875 0.076171875 4 1 196880 2 days, 6:41 1585272660000000000 pfSense.home 0.04296875 0.07666015625 0.0732421875 4 1 196890 2 days, 6:41 1585272670000000000 pfSense.home 0.03564453125 0.07568359375 0.0703125 4 1 196900 2 days, 6:41 1585272680000000000 pfSense.home 0.02978515625 0.07470703125 0.0673828125 4 1 196910 2 days, 6:41 1585272690000000000 pfSense.home 0.02490234375 0.07373046875 0.064453125 4 1 196920 2 days, 6:42 ...
How to drop influx measurement
bash-4.4# influx Connected to http://localhost:8086 version 1.8.3 InfluxDB shell version: 1.8.3 > auth username: admin password: > use pfsense Using database pfsense > drop measurement ip_block_log
What I updated: