In this lesson, we’ll talk about configuring the Screaming Frog Spider to Include and Exclude content based on rules you set.
Within the Configuration section of Screaming Frog, we have the option to instruct the Spider to include or exclude specific URLs and directories as we see fit. Using Regex (regular expressions), you can instruct the Spider where to go, and where not to go.
Adding the following to “Include” would tell the spider to only crawl URLs within the what-we-do directory. Likewise, adding this to the “Exclude” would omit these pages from the reports.
URL Contains Example
Adding the following to “Include” would tell the spider to only crawl URLs containing the word, “search.”. Likewise, adding this to the “Exclude” would omit these pages from the reports.
Using this feature is easiest with a quick course on Regex but some additional useful examples can be found here.
- For large sites, crawl small sections at a time
Rick Maggio brings over 15 years of tactical, hands-on experience managing digital advertising campaigns both small and large. Rick has worked in agency-land and on large in-house marketing teams and loves sharing his extensive experience with the LDA community.