Debugging a Sourcegraph Instance Without Grafana or Prometheus

Jason Harris

Updated November 03, 2023 17:49

Introduction

To monitor the health and performance of a Sourcegraph instance, adminis typically rely on Grafana and Prometheus, which provide detailed metrics and dashboards. However, there may be situations where access to Grafana or Prometheus is unavailable due to customer preferences. In such cases, it's important to know alternative methods for debugging a Sourcegraph instance.

In this article, we'll explore some of the key techniques and tools that can be used to debug a Sourcegraph instance when Grafana and Prometheus are not accessible.

Checking Sourcegraph Logs

Sourcegraph logs can provide valuable information about errors, warnings, and other events that may be affecting the system.

To view the logs for a specific Sourcegraph service, you can use the docker logs command if you're running Sourcegraph with Docker. For example, to view the logs for the frontend service, you can run the following command:

docker logs sourcegraph-frontend

If you're running Sourcegraph with Kubernetes, you can use the kubectl logs command to view the logs for a specific pod. For example, to view the logs for the frontend pod, you can run the following command:

kubectl logs sourcegraph-frontend

Note that a namespace (-n, --namespace) option may need to be added if the customer has their Sourcegraph instance deployed to a namespace in a cluster.

Examining System Resources

High resource (CPU, memory, disk space) usage can lead to performance issues and may indicate underlying problems with the system.

Since all Sourcegraph services are hosted with Linux, you can use standard Linux tools like top, htop, free, and df to monitor system resources. For example:

Use top or htop to view the current CPU and memory usage for all running processes. More info on top and htop.
Use free -h to view the total, used, and available memory on the system. More info on free
Use df -h to view the available disk space on the system. More info on the df command

Inspecting Network Traffic

Network issues can also impact the performance of a Sourcegraph instance. To debug network-related issues, you can use tools like ping, traceroute, curl, and tcpdump.

For example:

Use ping to check the network connectivity between the Sourcegraph instance and other services or hosts.
Use traceroute to identify the network path and any potential bottlenecks between the Sourcegraph instance and a target host.
Use curl to make HTTP requests to the Sourcegraph API or other endpoints to check their responsiveness.
Use tcpdump to capture and analyze network traffic to and from the Sourcegraph instance.

Reviewing Sourcegraph Configuration

Lastly, configuration issues can also lead to problems with a Sourcegraph instance. Be sure to review the Sourcegraph configuration files and environment variables to ensure that they are set correctly. Pay particular attention to settings related to database connections, external services, authentication, and resource limits.

The main Sourcegraph configuration file is typically located at /etc/sourcegraph/config.json (for Docker Compose deployments) or in a Kubernetes ConfigMap (for Kubernetes deployments). You can also use the Sourcegraph UI (profile > site-admin > site-config) to view and edit the configuration.

Conclusion

Keep in mind that the methods described in this article are not exhaustive, and there may be other tools and techniques that are relevant to your specific situation. Additionally, it's important to work closely with your team and communicate any findings or concerns, as collaboration is often key to successful troubleshooting.

Happy debugging!