We have quite a bit of services already running in our Build Your Homelab series, and with our SSL certificates up and running, we need to start monitoring our services. We have looked at Prometheus before (with the Grafana dashboard), so today we will start by getting our Prometheus Monitoring up and running. For Prometheus Monitoring to work, you will need to have Prometheus node exporter installed on the machines you need to monitor. Seeing as we are still going to install a number of services in the future, let’s first look at installing the Monitoring side of Prometheus, and then in due time, Grafana dashboards. We can then add the Prometheus node exporter on newly installed services and apps as we install them and just let the Monitoring side know to watch them. Anyway, let’s get started.
Step 1: Install Prometheus
We will use the amazing helper-scripts.com again to get a LXC container installed containing Prometheus. When searching for prometheus just make sure to select the correct one: https://community-scripts.github.io/ProxmoxVE/scripts?id=prometheus. Copy the command to get in installed from the section
To run this script, access your Proxmox instance. After logging in, expand the Datacenter node in the left navigation pane, then select the node you want to install Prometheus on. In the middle navigation pane, select 
. Paste the command from helper-scripts and press enter. Go through the steps:
On the first screen, you will be informed that you are about to create a new LXC container, select Yes to proceed.
On the Settings screen, we are going to select the default settings, as that is the recommended settings.
On the Storage Pools screen, select the disk where you want to install the LXC. If you only have one disk installed, the disk will automatically be selected and this screen will not be shown.
Wait for the installation to finish.
The container should be started. You will see a screen with the access details, so you can check that everything is working by accessing the address in a new tab. In our case it was http://192.168.1.18:9090
Step 2: Configure Prometheus
We now need to set up Prometheus to monitor our exported nodes. To do this, in the left navigation pane, select the newly created Prometheus LXC container. In the middle navigation pane, select 
. We now need to edit the configuration file. First, go to the directory:
cd /etc/prometheus/To edit the config file, run the command:
nano prometheus.ymlLook for the section at the bottom called static_configs, and add the nodes you want to monitor in the targets field, keeping in mind that the ports needs to be added (default is 9100).
Once done, save the config file (Ctrl+S) and exit the editor (Ctrl+X), then just click Shutdown, and then Reboot to reboot the Prometheus instance. 
Once rebooted, go back to the Prometheus dashboard, in our case http://192.168.1.18:9090. In the Query tab, you can run a few commands to confirm that your nodes are reporting. Here are some examples:
CPU Usage per core: rate(node_cpu_seconds_total{mode=~"system|user"}[1m])*100
Network Traffic Received: rate(node_network_receive_bytes_total[1m])
Network Traffic Sent: rate(node_network_transmit_bytes_total[1m])
Hard Drive Usage: 100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"} * 100) / node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"})
NGINX Config
If you want to add a reverse proxy for your Prometheus monitor in your NGINX VM, you can use the following config:
server {
listen 80;
server_name prometheus.local;
location / {
include proxy_params;
proxy_pass http://192.168.1.32:9090;
}
}Remember to change the IP address to the address of your Prometheus monitor LXC container. Also remember to add a record in your DNS for prometheus.local to point to the NGINX VM.
Conclusion
In conclusion, we’ve successfully laid the foundation for robust monitoring within our homelab by deploying Prometheus. This step marks a significant advancement in our ability to observe and understand the performance of our growing infrastructure. By leveraging the convenient helper-scripts.com, we streamlined the installation process, and configuring Prometheus to monitor our target nodes was straightforward. With the basic monitoring setup complete, we can now effectively track key metrics like CPU usage, network traffic, and disk utilization. This initial setup empowers us to proactively address potential issues and ensure the smooth operation of our services. In future installments, we’ll expand upon this foundation by integrating Grafana for enhanced visualizations and adding Prometheus node exporters to newly installed services, creating a comprehensive and scalable monitoring solution for our homelab.
