Hash Verification API v2

The Hash Verification API provides platforms with an interface to check if any content uploaded by their users has already been identified as terrorist content by the TCAP. Platforms query the API with the hash strings of their media files from their own systems. In turn, they receive a response indicating if any of those files are, or are likely to be, terrorist content.

Overview

What is the Hash Verification API

The API provides an endpoint that users can query with their own hash strings. These hash strings are then checked for exact or partial matches against the TCAP repository of hashed terrorist content and a response is returned indicating the results of these checks.

Key features and use cases

What hash algorithms are provided

We currently provide exact cryptographic hash-matching for the MD5, SHA256 and SHA512 algorithms as well as partial, or perceptual, hash matching for the PDQ (image) and TMK (video) algorithms.

How to use the Hash verification API v2

In order to use the API, you need to hash any content you wish to check against our archive. In the case of MD5, SHA256 and SHA512 this is fairly straightforward, however for TMK and PDQ you will likely have to fork Threat Exchange's code.

Endpoints

Verify cryptographic hashes

You can send a json list of items, where an item is of the following form for cryptographic hashes:

Endpoint

POST /hash-verification/api/v2

Request

Headers

  • Content-Type (Required: Yes): Must be application/json

Query Parameters

  • ideologies (Type: string, Required: No): <"far-right" | "islamist">

Request Body

This endpoints expects a list of items with the following fields

  • hash_value (Type: string, Required: Yes): A unique identifier representing either a PDQ hash string or a base-64 encoded TMK file.

  • hash_type (Type: string, Required: Yes): Specifies the format of the hash_value. It indicates whether the value is a PDQ hash or a TMK file.

  • confidence (Type: string, Required: No): A numerical value between 0 and 1 representing the level of certainty or reliability associated with the data represented by the hash

Example Request:

POST https://beta.terrorismanalytics.org/hash-verification/api/v2
{
  "body":[ 
    {
        "hash_value": "YOUR_HASH_STRING", 
        "hash_type": <"MD5" | "SHA256" | "SHA512">
    },
    ...
  ]
}
and the following for perceptual_hashes:
POST https://beta.terrorismanalytics.org/hash-verification/api/v2
{
  "body":[ 
    {
      "hash_value":  < "YOUR_PDQ_HASH_STRING" | base-64 encoded TMK file>,
      "hash_type": <"TMK" | "PDQ">
      "confidence": 0-1
    },
    ...
  ]
}

Filtering by ideology

You can include a query parameter, ideologies, if you want to restrict your search to a specific ideology. The options are far-right and islamist.

POST https://beta.terrorismanalytics.org/hash-verification/api/v2?ideologies=islamist

If you do not include the query parameter, all content will be searched regardless of ideology.

How many items can I send in a request?

For non-TMK hash verification requests, you can send up to 20 items in a list.

For TMK hash-verification requests, one item in a list. We may increase the number of TMKs per request in due course and, as such, have built the service to accept a list in the request.

Response

After receiving your list of items, we will match each one against our system and return a list with the same number of items, where an item is the following for cryptographic hashes (where True represents a match against content in our archive):

Status Code: 200

  • hash_value (Type: string): The calculated hash value of the data.
  • hash_type (Type: string): The algorithm used to generate the hash (e.g., MD5, SHA256, SHA512).
  • result (Type: boolean): Indicates whether a match was found for the provided hash.
  • error (Type: string or null): An optional error message if the request failed.

Example responses

200 OK
{
    "hash_value": "YOUR_HASH_STRING",
    "hash_type": <"MD5" | "SHA256" | "SHA512">,
    "result": <True/False>, 
    "error": <null | error message>
}
and the following for perceptual_hashes:
200 OK
{
    "hash_value":  < "YOUR_PDQ_HASH_STRING" | base-64 encoded TMK file>,
    "hash_type": <"TMK" | "PDQ">,
    "result": <True/False>, 
    "confidence": <0-1 | [0-1, 0-1], 
     "error": <null | error message>
}

Usage examples

Typescript example:

    const serviceUrl = "https://beta.terrorismanalytics.org/hash-verification/api/v2"
    const headers = {'Authorization': `Bearer ${token}`};
    const response = await fetch(service_url, {headers: headers});

Python example:

    service_url = "https://beta.terrorismanalytics.org/hash-verification/api/v2"
    headers = {'Authorization': f'Bearer {token}'}
    response = requests.get(serviceUrl, headers=headers)

Frequently Asked Questions

What are confidence scores?

While cryptographic hash verification simply involves checking your hash string for an exact match against our system, perceptual hashes check for similarity between two hashes and their respective pieces of content.

With this in mind, when you request a check on a cryptographic hash, you must include a confidence parameter between 0 and 1 where 1 is complete confidence, ie. an exact match.

We will return True in the result field if we have a piece of content whose hash matches or exceeds the confidence threshold you provided when compared with your hash_value. If we do not have any confidence that matches or exceeds this threshold, we will return False.

Please see our authentication documentation