- Joined
- Dec 30, 2024
- Messages
- 435
- Reaction score
- 219
- Points
- 62
- Website
- blackhatpakistan.net
- Points
- 1,378
- USD
- 1,378
A website checker is a mass status tool: feed it a list of URLs, it tells you which are alive, which are dead, which are behind challenges, and which are lying about their status through redirect chains and soft-404s. The naive version — fire a GET, read the status code, write live or dead — produces garbage at scale, because live/dead is not a status-code question: a 200 response on a parked-domain placeholder page is dead in every way that matters, a 403 from a WAF challenge is alive but protected, a 301 chain ending in a 200 through three hops needs all three hops resolved to judge the final destination, and a slow origin timing out at 10 seconds might be alive at 30. This is website-checker making decoded: the status semantics that actually classify live versus dead, the redirect and timeout discipline that keeps mass runs honest, the async worker architecture behind 10,000-URL batches, and the deployment stacks (Flask, PHP/XAMPP, CLI-first) that open tooling consistently lands on. Post your batch log — cleanest classification story gets pinned to Carding-method, first fifteen get the parked-page signature pack via PM.
TL;DR - The tool's job is classification, and the classification has more classes than live/dead. Alive-clean: final 2xx after redirect resolution, response body above the parked-page threshold, server headers present. Alive-protected: 403/401/429 or a challenge-page signature (Cloudflare's cf-chl, Incapsula, PerimeterX markers) — the site is up, access is gated; conflating this with dead is the single most common mass-checker error because it sends perfectly good domains to the dead file. Redirected-alive: one or more 3xx hops resolving to a live final destination — the record carries the full chain. Dead: connection refused, NXDOMAIN, definitive 404, or a 200 that fingerprints as parked (domain-squatter landing pages, "this domain is for sale" markers, empty-template signatures). Soft-dead: 5xx persisting past retry, origin timeouts past the retry budget, TLS handshake failures — likely dead or broken rather than parked, sorted separately because broken origins sometimes recover and parked domains never do. Unknown: network-layer errors from the checker's own side (local DNS failures, proxy-died mid-request) parked for re-run, never classified as dead on the tool's own failure. Mass architecture: async worker pool (hundreds of simultaneous in-flight requests per process, not thread-per-URL), two-knob timeouts (connect 5-10s, read 10-30s), one retry on timeout/5xx on a fresh connection, HEAD-first with GET fallback, domain-grouped scheduling so one host's URLs look like a single visitor session.
STATUS SEMANTICS THAT ACTUALLY WORK
- Method choice: HEAD first (half the bytes, the status and headers without the body), fall back to GET on targets that reject HEAD or return inconsistent HEAD/GET statuses — some origins 405 HEAD and 200 GET, some CDNs lie on HEAD; the two-step approach keeps mass throughput high while keeping classification honest for the minority of targets that misbehave on HEAD.
- Status-code mapping: 2xx is alive-candidate (awaiting body check), 3xx is follow (up to a hop cap, typically 5-10, with loop detection on repeated locations), 401/403 is alive-protected (record the auth-scheme header), 404/410 is dead, 429 is alive-rate-limited (site up, checker throttled — re-run class with backoff, not dead), 5xx is soft-dead-after-retry (one retry on a fresh connection before classifying), 000-class connection failures are refused (dead) or timeout (soft-dead-after-retry) by error type.
- Body-level truth: a 200 with a body under a threshold (a few hundred bytes) or matching parked-domain signatures ("domain is for sale", squatter-landing markers, template-empty heuristics) classifies parked-dead despite the 200 — the parked-page check is what stops domain-lists full of expired registrations from reading as fully alive; body checks only run on GET fallbacks, since HEAD carries no body.
- Challenge-page classification: Cloudflare, Incapsula, and PerimeterX challenge signatures (known marker strings and header patterns like cf-mitigated, server-challenge) classify alive-protected — the domain resolves, the origin answers, the edge is refusing unauthenticated clients; this class feeds its own output file because the consumer's next step (real browser, different network, timing) differs from dead-domain handling.
CODE: CLASSIFIER + HEAD/GET FLOW
The two-step method flow: HEAD first for throughput, GET fallback for classification honesty. The classifier maps status codes, resolves redirect chains with hop caps and loop detection, and runs the body-level parked check on GET responses. Unknown is never dead.
Code:
# website checker: HEAD-first flow + multi-class classifier
import aiohttp, asyncio, re
CONNECT_TIMEOUT = 8
READ_TIMEOUT = 15
HOP_CAP = 8
PARKED_SIGNS = [
"domain is for sale", "buy this domain", "sedoparking",
"hugedomains", "dan.com", "afternic", "this parked free",
"godaddy.com/domainsearch", # parked via godaddy
]
CHALLENGE_SIGNS = [
"cf-chl-", "cf_chl_opt", "challenge-platform",
"incapsula incident", "_Incapsula_Resource",
"px-captcha", "perimeterx", "captcha-delivery",
]
def classify(status, body, headers, chain_hops):
server = headers.get("Server", "").lower()
body_l = (body or "").lower()
if status == 000:
return "UNKNOWN_NET" # tool-side, never dead
if status in (401, 403):
for sign in CHALLENGE_SIGNS:
if sign in body_l or sign in server:
return "ALIVE_PROTECTED"
return "ALIVE_PROTECTED"
if status == 429:
return "RATE_LIMITED" # re-run class
if status in (404, 410):
return "DEAD"
if status == 200:
# body-level parked check
for sign in PARKED_SIGNS:
if sign in body_l:
return "PARKED_DEAD"
if len(body or "") < 300:
return "PARKED_DEAD" # empty-template heuristic
return "ALIVE_CLEAN"
if 300 <= status < 400:
if chain_hops >= HOP_CAP:
return "REDIRECT_LOOP"
return "REDIRECTED" # follow continues
if 500 <= status < 600:
return "SOFT_DEAD" # after one fresh retry
return "UNKNOWN"
async def check_url(session, url):
chain = []
current = url
for hop in range(HOP_CAP):
try:
async with session.head(current, allow_redirects=False,
timeout=aiohttp.ClientTimeout(
total=CONNECT_TIMEOUT + READ_TIMEOUT)) as r:
if r.status in (405, 501) or r.status >= 300:
# GET fallback for body check / redirect follow
async with session.get(current, allow_redirects=False,
timeout=aiohttp.ClientTimeout(
total=CONNECT_TIMEOUT + READ_TIMEOUT)) as r2:
body = await r2.text(errors="replace")
cls = classify(r2.status, body, dict(r2.headers), hop)
if cls == "REDIRECTED":
loc = r2.headers.get("Location", "")
if not loc:
return current, "REDIRECT_NO_LOC", chain
chain.append((r2.status, loc))
current = loc
continue
return current, cls, chain
else:
return current, classify(r.status, "", dict(r.headers), hop), chain
except (aiohttp.ClientConnectorError, asyncio.TimeoutError):
# retry once on fresh connection
try:
async with session.get(current, allow_redirects=False,
timeout=aiohttp.ClientTimeout(
total=CONNECT_TIMEOUT + READ_TIMEOUT)) as r:
body = await r.text(errors="replace")
return current, classify(r.status, body, dict(r.headers), hop), chain
except Exception:
return current, "SOFT_DEAD", chain
return current, "REDIRECT_LOOP", chain
print("classifier: HEAD-first, GET fallback, parked-body check, challenge-sig detect")THE MASS ARCHITECTURE
- Async worker model: an async HTTP client with a worker pool — not thread-per-URL (10,000 threads is a fantasy), but an async loop with a concurrency cap (hundreds of simultaneous in-flight requests per process, tuned against target politeness and the checker's own bandwidth); the queue drains the input list, each worker runs the two-step method flow (HEAD, GET fallback), classifies, writes to its class file with minimal lock contention (per-class buffered writers, not per-line file locks).
- Timeout budget: connect timeout and read timeout as separate knobs — connect around 5-10 seconds, read 10-30 depending on target class; the timeout choice directly decides soft-dead volume: aggressive timeouts flood the soft-dead file with slow-but-alive sites, lenient timeouts stall the queue behind dead origins; the two-timeout split lets connect-failures classify fast (dead/refused) while slow-reading origins get their full read budget.
- Retry policy: one retry on timeout and 5xx classes only, on a fresh connection; never retry 2xx/3xx/4xx classifications (they already have ground truth) and never retry through the same dead proxy (network-layer failures re-run on the worker's next proxy binding); the retry budget is what separates soft-dead (tried twice, still failing) from unknown (tool-side failure, re-run).
- Proxy and egress strategy: mass checking through shared egress gets rate-limited (429-class) or IP-banned at edge CDNs fast; the working patterns: residential proxy pool with per-worker binding for sensitive targets, direct egress with per-target pacing for politeness-sensitive batches, and domain-grouped scheduling (same-host URLs batched through the same worker so the checker's requests to one host look like a single visitor session, not a distributed hammer).
CODE: DOMAIN-GROUPED SCHEDULER + CLASS-FILE WRITER
URLs group by host before queueing so one host's requests batch through the same worker; class-file writers buffer per class to minimize lock contention at completion.
Code:
# domain-grouped scheduler + buffered class-file writers
from urllib.parse import urlparse
from collections import defaultdict
import threading
CLASS_FILES = ["alive_clean", "alive_protected", "redirected", "dead",
"parked_dead", "soft_dead", "rate_limited", "unknown"]
class ClassWriters:
def __init__(self, out_dir):
self.buffers = {c: [] for c in CLASS_FILES}
self.lock = threading.Lock()
self.out_dir = out_dir
def write(self, cls, line):
with self.lock:
if cls in self.buffers:
self.buffers[cls].append(line)
def flush(self):
import os
for cls, lines in self.buffers.items():
path = os.path.join(self.out_dir, cls + ".txt")
with open(path, "w", encoding="utf-8") as f:
f.write("\n".join(lines))
return {c: len(l) for c, l in self.buffers.items()}
def group_by_domain(urls):
"""Batch same-host URLs so one host's requests look like one visitor."""
groups = defaultdict(list)
for url in urls:
host = urlparse(url).netloc.lower()
groups[host].append(url)
# schedule: interleave hosts so no single host dominates a worker
queues = defaultdict(list)
max_len = max(len(v) for v in groups.values()) if groups else 0
for i in range(max_len):
for host, urls_list in groups.items():
if i < len(urls_list):
queues[host].append(urls_list[i])
return queues
# normalize + dedup before grouping
def normalize_url(url):
if not url.startswith("http"):
url = "https://" + url
p = urlparse(url)
path = p.path.rstrip("/") or "/"
return f"{p.scheme}://{p.netloc.lower()}{path}"
urls = ["https://a.com/x", "http://A.com/x/", "https://a.com/y", "https://b.com/z"]
normalized = list(set(normalize_url(u) for u in urls))
queues = group_by_domain(normalized)
print("unique:", len(normalized), "| host groups:", {h: len(v) for h, v in queues.items()})DETECTION LAYER
What runs against a mass website checker: edge-CDN reputation scoring (429-class spikes and IP bans on shared egress after throughput thresholds); challenge-page serving on reputation-flagged ranges (Cloudflare/Incapsula/PerimeterX markers appearing mid-run when the egress draws the edge's eye); rate-limit detection at origin (per-IP request budgets on politeness-sensitive targets); redirect-loop traps on parked-domain networks (squatter pages redirecting through chains that waste the hop cap); and soft-404 fingerprinting at the edge (200 responses on non-existent paths served by CDNs as placeholder pages). The detection surface is the CDN's own reputation stack plus origin-side rate counters — both see the request pattern, which is why domain-grouped scheduling and egress diversity keep mass runs honest.
OPSEC CHECKLIST
- Never thread-per-URL: async worker pools with a concurrency cap (hundreds in-flight per process), not one thread per target — 10,000 threads exhausts file descriptors and memory before the first classification batch completes.
- Two-knob timeouts from the start: connect 5-10s (decides refused-vs-dead fast), read 10-30s (gives slow-reading alive origins their budget) — aggressive single timeouts flood the soft-dead file, lenient ones stall the queue behind dead origins.
- HEAD-first with GET fallback: the two-step flow keeps throughput high while keeping classification honest for the minority of targets that misbehave on HEAD — some origins 405 HEAD and 200 GET, some CDNs lie on HEAD.
- Unknown is never dead: tool-side failures (local DNS errors, proxy-died mid-request) park in unknown.txt for re-run after egress rotation — classifying them dead poisons the dead file with recoverable targets.
- Domain-grouped scheduling: same-host URLs batched through one worker so the checker looks like a single visitor to each host, not a distributed hammer — the politeness discipline that keeps direct-egress batches running without origin-side bans.
- Output records carry the full story: final URL after redirects, redirect chain count and hop list, final status code, classification class, response time, content-length, server header, challenge-type marker, retry count — thin records ("url: dead") are useless at scale because consumers need the chain and the latency to debug and decide.
TEST YOURSELF: SIX QUESTIONS
Score honestly: (1) name the eight classification classes and which two get conflated most often by naive checkers. (2) what does the body-level parked check catch that a status-code-only checker misses, and why does it matter on expired-domain lists? (3) a target returns 405 on HEAD and 200 on GET — what does the two-step flow do, and why is this better than HEAD-only? (4) your 5,000-URL batch shows a sudden 100% timeout streak mid-run — dead internet or dead egress, and what does the unknown class do with it? (5) how does domain-grouped scheduling change the request pattern a single host sees from the checker? (6) a 200 response body is 180 bytes with "this domain is for sale" in the text — what class, and what would a status-code-only checker report? Zero to one: you read the TL;DR. Two to three: re-run your last batch's soft-dead file through the retry policy and check how many drain on fresh connections. Four plus: post your scoreboard with class tallies and wall time.
POST YOUR LOG
Scoreboard format: URLs input, unique after dedup, alive-clean count, alive-protected count (with challenge-type markers), redirected-alive count (chain-length distribution), dead count, parked-dead count, soft-dead count, rate-limited count, unknown count (and second-pass drain), average latency per class, wall time, egress count. Field-log framing, own batches only. Cleanest classification story gets pinned to Carding-method, first fifteen posters get the parked-page signature pack via PM — the squatter-landing marker strings, empty-template heuristics, and challenge-page signature list for Cloudflare/Incapsula/PerimeterX as of Q4 2026.
FAQ
- Q: Why does a simple status-code checker produce garbage at scale?
A: Because live/dead is not a status-code question — 200s on parked squatter pages are dead, 403s and challenge pages are alive-protected, redirect chains need resolution to judge the final destination, and tool-side network errors are unknown rather than dead. Each of those classes needs its own handling downstream, which is why the classification taxonomy is the tool's real product and the status code is just one input to it.
- Q: What is a soft-404 and why does it matter?
A: A 200 response whose body fingerprints as a parked-domain placeholder ("domain is for sale", squatter-landing markers, empty-template signatures) — the status code says alive, the content says dead. At scale, expired domain lists are full of these, and a checker without body-level parked detection reports near-100% alive on lists that are mostly dead domains, which poisons every consumer decision downstream.
- Q: HEAD or GET for mass checking?
A: HEAD first — half the bytes, carries the status and headers needed for most classifications — with automatic GET fallback on targets that reject HEAD or return inconsistent HEAD/GET statuses. The two-step flow keeps throughput high while keeping classification honest for the minority of misbehaving targets.
- Q: How should timeouts be configured?
A: Connect and read as separate knobs: connect 5-10 seconds (decides refused-vs-dead fast), read 10-30 seconds (gives slow-reading alive origins their budget). Aggressive single timeouts flood the soft-dead file with slow-but-alive sites, lenient ones stall the queue behind dead origins — the split plus one fresh-connection retry on timeout/5xx is what cleanly separates soft-dead from unknown.
- Q: Why do challenge pages need their own class?
A: A Cloudflare or Incapsula challenge means the domain resolves and the origin answers — the site is alive, the edge is refusing unauthenticated clients. Conflating protected with dead sends good domains to dead-file consumers (who drop or replace them), and the correct downstream action (real browser, different egress, timing the retry) is completely different from dead-domain handling.
- Q: What concurrency is safe for mass checking?
A: Async worker pools with a concurrency cap (hundreds in-flight per process, not thread-per-URL), per-target politeness via domain-grouped scheduling (one host's URLs batched through one worker so the checker looks like a single visitor), and egress strategy matching target sensitivity (residential proxies for CDN-protected batches, direct with pacing for politeness-sensitive targets). 429-class spikes mid-run mean egress reputation, not target death.
RELATED
How CC Checkers Actually Work - Response Codes, CPM and Accuracy
OpenBullet Config Making 2026 - Complete Guide to Creating Configs
Proxies for Carding 2026 - Complete Proxy Guide
CVV vs Fullz vs Logs - The Complete Material Guide
