Anchore can expose a variety of metrics to help you monitor performance and resource use. These metrics include number of images queued for analysis, duration of the async processes as they execute on a duty cycle, available space in the “tmp_dir” location for each container, and memory consumption for each Anchore service process. Monitoring these metrics can help ensure your Anchore deployment is resourced appropriately and running efficiently. Metrics are not enabled by default.
This article covers how to install Anchore Enterprise with metrics enabled as well as how to configure Prometheus to monitor Anchore Enterprise. The instructions in this article assume you have a running Kubernetes cluster, kubectl, and helm (v3) available.
Install Anchore Enterprise with metrics enabled
Follow these steps to install Anchore Enterprise using the Anchore Helm chart.
1. Create a namespace for our Anchore Enterprise secrets:
kubectl create namespace anchore
2. Create a secret for our Anchore Enterprise license:
kubectl create secret generic anchore-enterprise-license --from-file='license.yaml=<PATH/TO/LICENSE.YAML>' --namespace anchore
3. Create a secret for pulling the Anchore Enterprise images from the private DockerHub repository:
kubectl create secret docker-registry anchore-enterprise-pullcreds --docker-server='docker.io' --docker-username='<DOCKERHUB_USER>' --docker-password='<DOCKERHUB_PASSWORD>' --docker-email='<EMAIL_ADDRESS>' --namespace anchore
4. Add the Anchore Helm chart repository:
helm repo add anchore https://charts.anchore.io
5. Finally, install Anchore Enterprise using this custom anchore_values.yaml file, which provides overrides for enabling metrics generation. Note: This is not a production ready config, which typically would include external PostgreSQL and Redis services, along with resource sizing (see Guide to Deploying Anchore Enterprise on Kubernetes).
1.14.3 is the latest available chart version at the time this article was written. Feel free to use any compatible chart version
helm upgrade anchore anchore/anchore-engine --version 1.14.3 --install --namespace anchore --values anchore_values.yaml
After a few moments, you should see the Anchore pods running.
kubectl get pods --namespace anchore
NAME READY STATUS RESTARTS AGE
anchore-anchore-feeds-db-86957fdcdf-8dlww 1/1 Running 0 50s
anchore-postgresql-86dcfc86f9-cstcr 1/1 Running 0 50s
anchore-anchore-engine-enterprise-ui-5c49ddfd96-bq6jg 1/1 Running 0 50s
anchore-anchore-engine-policy-74df5c5fcd-vvt4d 1/1 Running 0 50s
anchore-anchore-engine-simplequeue-5f69b85b75-k9bqw 1/1 Running 0 50s
anchore-anchore-engine-catalog-5b584877f6-4lg6m 1/1 Running 0 50s
anchore-anchore-engine-analyzer-5bff8564d7-2x95z 1/1 Running 0 50s
anchore-anchore-engine-enterprise-feeds-849d699b56-8d7mt 1/1 Running 0 50s
anchore-anchore-engine-api-6f76744958-kmb98 5/5 Running 0 50s
anchore-anchore-ui-redis-master-0 1/1 Running 0 50s
Note: If you are seeing ImagePullBackoff errors, confirm that the anchore-enterprise-pullcreds secret contains the correct data. Similarly, if you are seeing CrashLoopBackoff errors, check that the anchore-enterprise-license secret was populated with the correct file. By default, the Helm chart will look for secrets with these names, but they can also be overridden via .Values.anchoreEnterpriseGlobal.imagePullSecretName and .Values.anchoreEnterpriseGlobal.licenseSecretName.
Install and configure Prometheus
In this example we will use the Kube Prometheus Stack to install Prometheus. The Kube Prometheus Stack includes Grafana, Alertmanager, and other useful components. The overrides file used in Step 2 uses Kube Prometheus Stack’s built-in method for creating pod and service monitors with appropriate namespace, labels, port names, and container paths. You can create these Kubernetes resources manually if desired.
1. Add the Prometheus Community Helm chart repository.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2. Install the Kube Prometheus Stack using this custom prometheus_values.yaml file, which provides overrides for creating pod and service monitors for scraping metrics from the Anchore Enterprise pods and services. Note: This is not a production ready config. Please see the Prometheus documentation for more info.
18.0.2 is the latest available chart version at the time this article was written—feel free to use any compatible chart version.
helm upgrade prometheus prometheus-community/kube-prometheus-stack --version 18.0.2 --install --namespace prometheus --create-namespace --values prometheus_values.yaml
After a few moments, you should see the Kube Prometheus Stack pods running.
kubectl get pods --namespace prometheus
NAME READY STATUS RESTARTS AGE
prometheus-prometheus-node-exporter-4c2qc 1/1 Running 0 23s
prometheus-prometheus-node-exporter-qrq9t 1/1 Running 0 23s
prometheus-prometheus-node-exporter-wgfr6 1/1 Running 0 23s
prometheus-prometheus-node-exporter-grxr5 1/1 Running 0 23s
prometheus-kube-prometheus-operator-75ffb464df-srgnh 1/1 Running 0 23s
prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 19s
prometheus-kube-state-metrics-76f66976cb-4dlj4 1/1 Running 0 23s
prometheus-grafana-774b554f4d-tvqrw 2/2 Running 0 23s
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 19s
Monitor Anchore Enterprise with Prometheus, Grafana, and Alertmanager
With the Kube Prometheus Stack installed and configured, you can add a Grafana dashboard for visualizing Anchore Enterprise metrics in real time.
1. Verify that the Prometheus targets are up using kubectl’s port-forward command. Note that port-forward is used here for simplicity, but in a production environment you should expose the service with an ingress.
kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090 --namespace prometheus
2. Navigate to http://localhost:9090/targets in a browser to access the Prometheus Targets page. You should see the Anchore pod and service monitors showing "(1/1 up)".
3. Once you have verified that the pod and service monitors are healthy, access the Grafana dashboard in a similar fashion.
sudo kubectl port-forward svc/prometheus-grafana 80 --namespace prometheus
4. When you navigate to http://localhost:80 in a browser, you should see the “Welcome to Grafana” login page. Enter the default Grafana username:password (admin:prom-operator) to log into Grafana. Navigate to http://localhost:80/dashboard/import and you will be prompted to either upload or paste a Grafana dashboard. Upload the anchore-general-grafana-dashboard.json file and click “Import”. You will be taken to the dashboard where you should start to see metrics.
5. Now that you can visualize Anchore Enterprise metrics in real time, you can start configuring Alertmanager to notify your team when something isn’t healthy or a certain threshold is reached. To access the Alertmanager dashboard, simply use the kubectl port-forward command and navigate to http://localhost:9093.
kubectl port-forward svc/prometheus-kube-prometheus-alertmanager 9093 --namespace prometheus
You should now be well on your way to monitoring Anchore Enterprise. If you need additional assistance, please do hesitate to contact our support team.