How to Selectively Crawl URLs with Screaming Frog

Screaming Frog, like other crawl/audit tools, provides multiple options to crawl a website. The most common and obvious way to use Screaming Frog is by entering your website's URL in Spider mode. For some time, this was the option I used the most.

However, there are situations when you may need to, for example, just crawl a single URL to work on it — or even crawl a group of URLs, such as specific blog articles. So now you may be wondering how to crawl specific URLs on Screaming Frog, right? Let's dig into it with some practical examples.

How to Crawl a Single URL with Screaming Frog

How do I crawl just one URL in Screaming Frog?

To crawl a single URL with Screaming Frog, follow these steps:

1. Check that Screaming Frog is in Spider Mode. Go to Mode > Spider in the top menu.

1. Next to the URL input bar, click the dropdown and select Exact URL.

1. Paste or type the URL you want to crawl.
2. Click Start to begin the crawl.

Screaming Frog will audit only that specific URL, which is useful when you want to focus on a single page and avoid pulling in data from the rest of the site. Instead of crawling the entire website and then applying filters, this method is quicker and uses less memory.

How to Crawl a Group of URLs with Screaming Frog

How can I crawl multiple specific URLs in Screaming Frog?

Sometimes, you may need to crawl a group of URLs — for example, to analyze a series of blog posts within a category.

To do this, Screaming Frog offers a List Mode:

1. Go to Mode > List in the top menu.

1. Click the Upload dropdown next to the URL input bar.

1. Select one of the following options:
   • From a File: Upload a .txt, .csv, or .xlsx file with one URL per line.
   • Enter Manually: Opens a box where you can type or paste a list of URLs.
   • Paste: Automatically pastes the URLs currently in your clipboard.
   • Download XML Sitemap: Provide the URL of your sitemap. Screaming Frog will crawl only the URLs included in that sitemap. If your website is built in WordPress and you're using the Yoast SEO plugin, you can crawl your posts' sitemap or pages' sitemap independently (e.g., /post-sitemap.xml).

This feature is perfect for focused audits or SEO checks on content clusters.

How to Filter URLs in Spider Mode

Can I limit what Screaming Frog crawls in Spider Mode?

Yes. You can apply filters using the Configuration menu:

• Configuration > Include: Crawl only URLs that match specific patterns.
• Configuration > Exclude: Prevent certain folders, subdomains, or URLs from being crawled.

These filters allow for more precise and efficient crawls when working in Spider Mode. If your website is large and contains hundreds or even thousands of URLs, you may find these options particularly useful. During my career, I've worked with older websites that gradually accumulated thousands of URLs related to posts, events, uploads, and more.

Additionally, a general crawl in Spider Mode will also include JS and CSS files, which are often not necessary for SEO audits.

By using the Exclude option in the Crawl Configuration, you can omit certain file extensions or, even better, entire subfolders — such as "wp-includes" or "uploads" in the case of WordPress.

Summary

If you need to crawl just one page or a selected group of URLs with Screaming Frog, you don’t have to scan the entire website. Use Exact URL mode for single pages, List mode for multiple URLs, and XML sitemap downloads for structured, targeted crawls. Filtering through Include or Exclude configurations gives you even more control over what Screaming Frog processes.

These features make Screaming Frog a flexible and powerful tool for targeted SEO audits and troubleshooting.