FoxTrust pre-alpha

Documentation

Three ways to use FoxTrust. All of them give the same customer verdict: a risk from 0 to 100, a level and the reasons, each with its code, when it was last seen and how much it adds.

API

Look up one address per request over HTTPS. Keys, free-tier limits, answer fields and error codes.

Signed snapshots

The whole data set as MaxMind DB files, a delta every hour and a full file every day. How to download them, verify their signatures and read them with any MMDB reader.

/verify

A forward-auth service that runs next to your reverse proxy (nginx, Traefik or Caddy), reads the signed snapshots and applies your policy to each request: allow, challenge with a proof-of-work page instead of a captcha, or block. Its setup guide is being written for this site.

Analytics. Set FOXTRUST_ACCOUNT_KEY to a key from your account, and /verify sends counts of its decisions once a minute. Your account then shows them under Analytics: requests over time by action, the rules that fired, the challenge funnel, countries and networks, levels and reason codes.

Logs. /verify writes visitor addresses to its log as their /24 or /48 network.

Search crawlers

Search crawlers cannot solve a challenge, so a policy that challenges them takes your site out of search results. Search crawler addresses carry the category search_crawler: addresses in the ranges that search engines publish for their crawlers, and crawler addresses that FoxTrust confirms by reverse DNS for a search engine that publishes no list. It adds nothing to the risk; it only names the address. Put a rule that allows it before every other rule, whether you write your policy for /verify or apply the API or the snapshots yourself:

rules:
  - name: search-crawlers
    when:
      categories: [search_crawler]
    action: allow
  # your other rules follow

Search crawlers in the address ranges that search engines publish for their crawlers, and crawler addresses that FoxTrust confirms by reverse DNS, are never challenged or blocked by a policy that puts the search_crawler rule first. This does not cover other crawler addresses, a list its search engine stopped updating, or search engines that FoxTrust does not check. policy check and the /status page of /verify warn when the rule is missing or comes after a rule on hosting, cloud or a level.

AI crawlers and agents

AI companies publish the addresses of their bots. FoxTrust marks those addresses by what the company says the bot is for. Like search_crawler, these categories add nothing to the risk; they only name the address:

The example policy has no rule for them, so they are treated like any other address until you decide. The first rule that matches an address decides, so the order of the rules matters. An address can carry more than one of these categories when its company uses it for several purposes. Keep the search crawler rule first in every recipe.

Refuse training, strictly. An address that is used for both training and AI search is blocked:

rules:
  - name: search-crawlers
    when:
      categories: [search_crawler]
    action: allow
  - name: no-ai-training
    when:
      categories: [ai_training]
    action: block
  - name: ai-search
    when:
      categories: [ai_search]
    action: allow
  # your other rules follow

Prefer AI search. An address that is used for both is allowed:

rules:
  - name: search-crawlers
    when:
      categories: [search_crawler]
    action: allow
  - name: ai-search
    when:
      categories: [ai_search]
    action: allow
  - name: no-ai-training
    when:
      categories: [ai_training]
    action: block
  # your other rules follow

To make agents acting for a person prove they are welcome, add a rule that challenges ai_user.

What these categories cannot do:

Ads and other special crawlers

A search engine also runs crawlers that are not search crawlers: they check the landing pages of ads, fetch pages for the ads shown on them, fetch for its APIs, and scan links for abuse. Their addresses carry the category ads_crawler, from the ranges the search engine publishes for them, confirmed by reverse DNS where the addresses have reverse names. It adds nothing to the risk and is not search_crawler: these crawlers ignore the rules that robots.txt gives to every crawler (User-agent: *) and follow only rules that name them, and the abuse scanner ignores robots.txt altogether.

The example policy has no rule for them. If you advertise and a challenge on your landing pages would get your ads disapproved, allow them after the search crawler rule:

rules:
  - name: search-crawlers
    when:
      categories: [search_crawler]
    action: allow
  - name: ads-crawlers
    when:
      categories: [ads_crawler]
    action: allow
  # your other rules follow

Bots FoxTrust does not mark

Some bots that you may want to let through carry no category, because no published list identifies them by address:

Your own address

My IP shows the address you are browsing from and what FoxTrust's data says about it, with each reason explained. For scripts, https://foxtrust.dev/ip answers with the address as plain text and https://foxtrust.dev/ip.json with {"ip": "…", "family": 4}; use curl -4 or curl -6 to choose the family. They need no key, give no verdict and take no parameters, and allow at most 60 requests a minute from one address.