Playwright 爬虫技能：高级网页抓取与反爬突破 - Openclaw Skills

什么是 Playwright 爬虫技能？

Playwright 爬虫技能是为 Openclaw Skills 生态系统构建的多功能数据提取引擎。它为开发者提供了分层抓取方案，允许根据目标网站的安全级别选择最有效的抓取方式。从简单的静态获取到绕过复杂的 Cloudflare 挑战，该技能确保您的 AI 代理能够访问所需的网页数据而不会被封锁。

通过利用 Playwright 的强大功能，该技能比传统的抓取库更有效地模拟人类浏览器行为。它包含了用于隐身操作的专门脚本，使其成为研究人员、开发者和数据分析师在 Openclaw Skills 集合中获取动态或受保护网页内容时不可或缺的工具。

下载入口:https://github.com/openclaw/skills/tree/main/skills/waisimon/playwright-scraper-skill

安装与下载

1. ClawHub CLI

从源直接安装技能的最快方式。

npx clawhub@latest install playwright-scraper-skill

2. 手动安装

将技能文件夹复制到以下位置之一

全局模式 ~/.openclaw/skills/ 工作区 <project>/skills/

优先级：工作区 > 本地 > 内置

3. 提示词安装

将此提示词复制到 OpenClaw 即可自动安装。

请帮我使用 Clawhub 安装 playwright-scraper-skill。如果尚未安装 Clawhub，请先安装（npm i -g clawhub）。

Playwright 爬虫技能 应用场景

• 从需要完整浏览器渲染的重 JavaScript 动态网站提取内容。
• 绕过社区论坛和新闻网站的高级反爬保护及 Cloudflare 验证。
• 在 Openclaw Skills 工作流中自动收集结构化数据，用于竞争分析和研究。
• 通过自动截图和 HTML 存档生成页面状态的视觉证据。

Playwright 爬虫技能 工作原理

1. 用户确定目标 URL 并评估目标站点的反爬级别。
2. 技能选择合适的执行路径：简单站点使用内置 fetch，JS 密集型站点使用 Playwright Simple，受保护站点使用 Playwright Stealth。
3. 在隐身模式下，技能初始化一个隐藏了自动化特征（如 navigator.webdriver 属性）的浏览器实例。
4. 会话中应用了真实的 User-Agent 和类人交互模式（如随机延迟）。
5. 浏览器导航至目标，等待 DOM 或网络进入空闲状态，然后提取页面内容。
6. 处理后的数据作为结构化 JSON 对象返回给 Openclaw Skills 环境，包含元数据和可选的截图文件路径。

Playwright 爬虫技能 配置指南

要将此功能集成到您的 Openclaw Skills 设置中，请遵循以下安装步骤：

cd playwright-scraper-skill
npm install
npx playwright install chromium

Playwright 爬虫技能 数据架构与分类体系

Playwright 爬虫技能将其输出组织成结构化格式，以确保与其他 Openclaw Skills 的兼容性。

属性	类型	描述
url	字符串	被抓取的目标 URL。
title	字符串	从元数据中提取的页面标题。
content	字符串	页面的主要文本或 HTML 内容。
elapsedSeconds	字符串	执行抓取所花费的总时间。
screenshot	路径	（可选）生成的 PNG 截图的本地路径。
html_file	路径	（可选）保存的原始 HTML 源码的本地路径。

name: playwright-scraper-skill
description: Playwright-based web scraping OpenClaw Skill with anti-bot protection. Successfully tested on complex sites like Discuss.com.hk.
version: 1.2.0
author: Simon Chan

Playwright Scraper Skill

A Playwright-based web scraping OpenClaw Skill with anti-bot protection. Choose the best approach based on the target website's anti-bot level.

---

?? Use Case Matrix

---

?? Installation

cd playwright-scraper-skill
npm install
npx playwright install chromium

---

?? Quick Start

1?? Simple Sites (No Anti-Bot)

Use OpenClaw's built-in web_fetch tool:

# Invoke directly in OpenClaw
Hey, fetch me the content from https://example.com

---

2?? Dynamic Sites (Requires JavaScript)

Use Playwright Simple:

node scripts/playwright-simple.js "https://example.com"

Example output:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "content": "...",
  "elapsedSeconds": "3.45"
}

---

3?? Anti-Bot Protected Sites (Cloudflare etc.)

Use Playwright Stealth:

node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

Features:

• Hide automation markers (navigator.webdriver = false)
• Realistic User-Agent (iPhone, Android)
• Random delays to mimic human behavior
• Screenshot and HTML saving support

---

4?? YouTube Video Transcripts

Use deep-scraper (install separately):

# Install deep-scraper skill
npx clawhub install deep-scraper

# Use it
cd skills/deep-scraper
node assets/you@tube_handler.js "https://[email protected]/watch?v=VIDEO_ID"

---

?? Script Descriptions

scripts/playwright-simple.js

• Use Case: Regular dynamic websites
• Speed: Fast (3-5 seconds)
• Anti-Bot: None
• Output: JSON (title, content, URL)

scripts/playwright-stealth.js ?

• Use Case: Sites with Cloudflare or anti-bot protection
• Speed: Medium (5-20 seconds)
• Anti-Bot: Medium-High (hides automation, realistic UA)
• Output: JSON + Screenshot + HTML file
• Verified: 100% success on Discuss.com.hk

---

?? Best Practices

1. Try web_fetch First

If the site doesn't have dynamic loading, use OpenClaw's web_fetch tool—it's fastest.

2. Need JavaScript? Use Playwright Simple

If you need to wait for JavaScript rendering, use playwright-simple.js.

3. Getting Blocked? Use Stealth

If you encounter 403 or Cloudflare challenges, use playwright-stealth.js.

4. Special Sites Need Specialized Skills

• YouTube → deep-scraper
• Reddit → reddit-scraper
• T@witter → bird skill

---

?? Customization

All scripts support environment variables:

# Set screenshot path
SCREENSHOT_PATH=/path/to/screenshot.png node scripts/playwright-stealth.js URL

# Set wait time (milliseconds)
WAIT_TIME=10000 node scripts/playwright-simple.js URL

# Enable headful mode (show browser)
HEADLESS=false node scripts/playwright-stealth.js URL

# Save HTML
SAVE_HTML=true node scripts/playwright-stealth.js URL

# Custom User-Agent
USER_AGENT="Mozilla/5.0 ..." node scripts/playwright-stealth.js URL

---

?? Performance Comparison

---

??? Anti-Bot Techniques Summary

Lessons learned from our testing:

? Effective Anti-Bot Measures

1. Hide navigator.webdriver — Essential
2. Realistic User-Agent — Use real devices (iPhone, Android)
3. Mimic Human Behavior — Random delays, scrolling
4. Avoid Framework Signatures — Crawlee, Selenium are easily detected
5. Use addInitScript (Playwright) — Inject before page load

? Ineffective Anti-Bot Measures

1. Only changing User-Agent — Not enough
2. Using high-level frameworks (Crawlee) — More easily detected
3. Docker isolation — Doesn't help with Cloudflare

---

?? Troubleshooting

Issue: 403 Forbidden

Solution: Use playwright-stealth.js

Issue: Cloudflare Challenge Page

Solution:

1. Increase wait time (10-15 seconds)
2. Try headless: false (headful mode sometimes has higher success rate)
3. Consider using proxy IPs

Issue: Blank Page

Solution:

1. Increase waitForTimeout
2. Use waitUntil: 'networkidle' or 'domcontentloaded'
3. Check if login is required

---

?? Memory & Experience

2026-02-07 Discuss.com.hk Test Conclusions

• ? Pure Playwright + Stealth succeeded (5s, 200 OK)
• ? Crawlee (deep-scraper) failed (403)
• ? Chaser (Rust) failed (Cloudflare)
• ? Puppeteer standard failed (403)

Best Solution: Pure Playwright + anti-bot techniques (framework-independent)

---

?? Future Improvements

• Add proxy IP rotation
• Implement cookie management (maintain login state)
• Add CAPTCHA handling (2captcha / Anti-Captcha)
• Batch scraping (parallel URLs)
• Integration with OpenClaw's browser tool

---

?? References

• Playwright Official Docs
• puppeteer-extra-plugin-stealth
• deep-scraper skill