Why files are skipped from indexing

Why files are skipped from indexing

Overview

Sourcegraph indexes code on the default branch of each repository to speed up searches across many repositories. Not every file is eligible for indexed search. This article explains why files may be skipped, how administrators can identify the reason, and what to do next.

Common reasons files are skipped

• File size: By default, files larger than 1 MB are excluded.

• Binary content: Binary files are not indexed.

• Trigram limit: Files with more than 20,000 unique trigrams—sequences of three characters—are skipped.

• Invalid encoding: Files that are not valid UTF-8 are skipped.

How to see why a file was skipped

Admin access is required.

1. Open the repository in Sourcegraph.

2. Go to Repository settings → Indexing.

3. Click the N skipped files link.

4. In the generated search query, remove select:file to display content results.

5. Look for the NOT-INDEXED reason, such as NOT-INDEXED: exceeds the maximum size limit or an encoding-related message. The exact wording may vary by reason and Sourcegraph version.

What to do next

• Large text files: Add the file path or a glob pattern to the search.largeFiles setting in site configuration, then reindex the repository. This setting can force eligible text files, such as yarn.lock, to be included regardless of size.

• Binary files: Binary files remain excluded even when they match search.largeFiles.

• Invalid UTF-8: Convert the file to valid UTF-8. Files must still be valid UTF-8 even when they match search.largeFiles.

Further reading

See Indexed search in the Sourcegraph documentation.