How To Crawl Your Entire Site With JavaScript And UTM Parameters Using Crawlbase On MacOS

SHARE THIS NOW!

illphated

How to Crawl Your Entire Site with JavaScript and UTM Parameters Using Crawlbase on macOS
At illphated.com, I like to automate everything—especially when it comes to tracking page performance and how different links behave. Recently, I needed a reliable way to crawl all pages of my site, including those with dynamic JavaScript and UTM parameters, for SEO and analytics research.

Enter Crawlbase (formerly ProxyCrawl), a powerful scraping platform that supports real browser rendering—perfect for modern websites with JavaScript and multimedia.

Here’s how I automated crawling illphated.com from macOS using a simple shell script, including support for ?utm=1 through ?utm=999.

🔧 The Problem
Most scrapers choke on JavaScript-heavy pages or fail to simulate realistic browser traffic. I needed something that:

Renders JavaScript like a real browser

Handles multimedia and dynamic loading

Looks like a real user (not a bot)

Can cycle through all my UTM-tracked URLs

🚀 The Solution: Crawlbase + Shell Script
Using Crawlbase’s JavaScript token, I built a macOS script to crawl:

✅ Static pages like /about, /blog, etc.
✅ Dynamic URLs like /?utm=1 through /?utm=999
✅ Rendered JavaScript just like a real user would see
✅ A modern Windows Chrome user-agent to stay stealthy

🧠 The Script (Save as crawl_illphated.sh)

#!/bin/bash

# Crawlbase JavaScript Token
TOKEN=”GqmUjbLgg1HOFyfYIQnhlQ”
BASE_URL=”https://api.crawlbase.com”
SITE_URL=”https://illphated.com”

# Latest Windows Chrome user-agent (July 2025)
USER_AGENT=”Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.127 Safari/537.36″

# Output directory
OUTPUT_DIR=”./crawl_results”
mkdir -p “$OUTPUT_DIR”

# Encode URLs for the API
urlencode() {
local raw=”$1″
python3 -c “import urllib.parse; print(urllib.parse.quote(”’$raw”’, safe=”))”
}

# Static paths to crawl
STATIC_PATHS=(
“”
“/about”
“/blog”
“/contact”
“/travel”
“/search?q=cyberpunk”
)

# Crawl static pages
echo “🚀 Crawling static pages…”
for path in “${STATIC_PATHS[@]}”; do
FULL_URL=”${SITE_URL}${path}”
ENCODED_URL=$(urlencode “$FULL_URL”)
OUTPUT_FILE=”$OUTPUT_DIR/$(echo “$path” | tr ‘/?’ ‘__’ | sed ‘s/^_//’).html”

echo “🔍 Crawling: $FULL_URL”
curl -s -A “$USER_AGENT”
-H “Accept-Encoding: gzip”
“$BASE_URL/?token=$TOKEN&url=$ENCODED_URL&render=true”
-o “$OUTPUT_FILE”
sleep 0.25
done

# Crawl all ?utm=1 to ?utm=999 URLs
echo “🚀 Crawling UTM query pages…”
for i in $(seq 1 999); do
FULL_URL=”${SITE_URL}/?utm=$i”
ENCODED_URL=$(urlencode “$FULL_URL”)
OUTPUT_FILE=”$OUTPUT_DIR/utm_$i.html”

echo “🔍 Crawling: $FULL_URL”
curl -s -A “$USER_AGENT”
-H “Accept-Encoding: gzip”
“$BASE_URL/?token=$TOKEN&url=$ENCODED_URL&render=true”
-o “$OUTPUT_FILE”
sleep 0.25
done

echo “🎉 Full crawl complete! Results saved to $OUTPUT_DIR”

🧪 How to Run It on macOS
Save the script as crawl_illphated.sh

Make it executable:

bash
Copy
Edit
chmod +x crawl_illphated.sh
Run it:

bash
Copy
Edit
./crawl_illphated.sh
It will create a folder called crawl_results/ filled with rendered HTML files from every page crawled.

🎯 Why It’s Useful
This is perfect for:

Verifying UTM tracking link functionality

Benchmarking SEO and rendering speed

Testing dynamic elements or lazy-loaded content

Running offline snapshots of your website for archival or QA

👀 What’s Next?
Want to level it up?

Add link discovery to crawl deeper from internal pages