# Authentication

To use some TCAP API services, you will need to obtain a resource-specific, short-lived access token.

In order to get one of these, you will already need to be an on-boarded TCAP or TCAP Archive user with a username and password.

Services which currently utilise this authentication system:
- [Hash Verification API](https://terrorismanalytics.org/docs/hash-verifier-v2)

Authentication for other services is more straightforward, see [other docs](https://terrorismanalytics.org/docs)

## Obtaining an Access Token
To obtain a token make a request to the TCAP authentication endpoint with your username, password and the name of the requested service, for example `Hash Verification`:

```
POST https://beta.terrorismanalytics.org/token-auth/api/login
{
  username: YOUR_TCAP_USERNAME,
  password: YOUR_TCAP_PASSWORD,
  requested_service: NAME_OF_REQUESTED_API_SERVICE
}
```

## Using the token
Once you have received a token, you should attach it as a header to any API service request.

Typescript example:

```
    const serviceUrl = "https://beta.terrorismanalytics.org/hash-verification/api/v2"
    const headers = {'Authorization': `Bearer ${token}`};
    const response = await fetch(service_url, {headers: headers});

```

Python example:

```
    service_url = "https://beta.terrorismanalytics.org/hash-verification/api/v2"
    headers = {'Authorization': f'Bearer {token}'}
    response = requests.get(serviceUrl, headers=headers)

```

## How long does the token last?
At the moment, tokens are configured to be valid for 5 minutes. When a token expires, you will receive the following response with a status of `403`:

```
Authentication error: Token has expired. Please refresh.
```

You can request a new token anytime and continue using it to make requests to your chosen service.

If engaging with the service programmatically, we recommend writing a script to request a new token whenever you receive a response with body "Authentication error: Token has expired. Please refresh."

## Further help
Our dev team would be happy to walk you through the authentication process. 

If you wish to reach out in relation to this or anything else surrounding our API services, please contact us through [this form](https://terrorismanalytics.org/about/contact) 

Get started using the Tech Against Terrorism TCAP API. This document outlines how you can obtain an authentication token and use our API services.

Authenticating for TCAP API Services

## Two endpoints
The Submissions API consists of two endpoints for tech platform TCAP users. 

###Content endpoint
The first returns a list of urls marked as terrorist content by Tech Against Terrorism:

```
GET https://beta.terrorismanalytics.org/integrations/tech_platform_api/content
```

###Alerts endpoint
The second returns a list of alerts we have sent as well as information about the url associated with each alert:

```
GET https://beta.terrorismanalytics.org/integrations/tech_platform_api/content_alert
```

##Authentification
In order to hit either endpoint you will need to be an on-boarded TCAP user associated with a tech platform. These endpoints sit inside the main TCAP backend so you will be able to access both with a standard TCAP JWT token.

To obtain a token make a request to the TCAP authentication endpoint with your username and password.

```
POST https://beta.terrorismanalytics.org/token-auth/tcap/
{
  username: YOUR_TCAP_USERNAME,
  password: YOUR_TCAP_PASSWORD,
}
```

## Your platform's content
When  you send an authenticated request to either of these endpoints, the service will filter content or content alerts based on the tech platform associated with your email address. If your email address happens to be associated with more than one tech platform, it will return data associated with each platform. However, if you wish to refine the  search, you can include a query parameter specifying the tech platform, for example:

```
GET https://beta.terrorismanalytics.org/integrations/tech_platform_api/content_alert
?tech_platform=Platform C
```

or

```
GET https://beta.terrorismanalytics.org/integrations/tech_platform_api/content
?tech_platform=Platform C
```

##Refining your request
You can further refine your requests to either endpoint by including the following query paramaters:

- `created_on` : the date the URL was submitted to the TCAP.
- `terrorist_group`: the terrorist group that is associated to the content.

Here are some example urls:

```
GET https://beta.terrorismanalytics.org/integrations/tech_platform_api/content
?created_on=2023-06-29T10:03:54.828554

GET https://beta.terrorismanalytics.org/integrations/tech_platform_api/content_alert
?terrorist_group=Terrorist Group A


```

##Our Responses
###Content API response

The content endpoint returns the following data on each url that satisfies your query:

- `created_on` `(Date)` : Timestamp at which the URL was approved for submission to the TCAP

- `terrorist_group` `(String)`: Terrorist group associated with the content submitted to the TCAP

- `tech_platform` `(String)`: Platform where the content was collected from

- `tier` `(String)`: Tier of alert submitted for this URL (Threat To Life, Crisis, Designation, Promotional, Recommended)

- `md5_hash` `(String)`: MD5 hash encryption of the URL where the content was found

- `url_status` `("Online" | "Offline")`: Online status of the URL (online or offline)

- `extreme_content` `(Boolean)`: Whether the URL contains graphic and/or extreme content

- `pii` `(Boolean)`: Whether the URL contains personally identifiable information (PII)

- `content_alert` `(Date)`: Time/date when the tech platform was alerted for this URL


####What is the md5 hash?
The MD5 hash is a cryptographic hash of the url associated with the content. In essence it is the unique identifier of that piece of content. We are sending it in this format for security purposes.

###Content Alert API response

The content alert endpoint returns the following data on each successfully sent alert that satisfies your query:

- `content_item` `(ContentItem)`: all of the information from the Content API response above, nested

- `to` `(String)`: the email address the alert was sent to

- `alerted_date` `(Date)`: the date the alert was sent





The Submissions API provides two endpoints for tech platforms to query our database and find out what contents our OSINT team have found on their platforms as well what alerts we have issued to them. Platforms can refine their queries to specify dates and specific terrorist groups.

TCAP Submissions API (v1.2.0)

## What is the Hash Verification API
The API provides an endpoint that users can query with their own hash strings. These hash strings are then checked for exact or partial matches against the TCAP repository of hashed terrorist content and a response is returned indicating the results of these checks.


## What hash algorithms are provided
We currently provide exact cryptographic hash-matching for the MD5, SHA256 and SHA512 algorithms as well as partial, or perceptual, hash matching for the PDQ (image) and TMK (video) algorithms.


## How to use the Hash verification API v2
In order to use the API, you need to hash any content you wish to check against our archive. In the case of MD5, SHA256 and SHA512 this is fairly straightforward, however for TMK and PDQ you will likely have to fork [Threat Exchange's code](https://github.com/facebook/ThreatExchange/tree/main).


### Your requests
You can send a json list of items, where an item is of the following form for cryptographic hashes:

```
POST https://beta.terrorismanalytics.org/hash-verification/api/v2
{
  "body":[ 
    {
        "hash_value": "YOUR_HASH_STRING", 
        "hash_type": <"MD5" | "SHA256" | "SHA512">
    },
    ...
  ]
}

```

 and the following for perceptual_hashes:

```  
POST https://beta.terrorismanalytics.org/hash-verification/api/v2
{
  "body":[ 
    {
      "hash_value":  < "YOUR_PDQ_HASH_STRING" | base-64 encoded TMK file>,
      "hash_type": <"TMK" | "PDQ">
      "confidence": 0-1
    },
    ...
  ]
}

```

### Filtering by ideology
You can include a query parameter, `ideologies`, if you want to restrict your search to a specific ideology. The options are `far-right` and `islamist`.

```  
POST https://beta.terrorismanalytics.org/hash-verification/api/v2?ideologies=islamist
```

If you do not include the query parameter, all content will be searched regardless of ideology.


### How many items can I send in a request?
For non-TMK hash verification requests, you can send up to 20 items in a list.

For TMK hash-verification requests, one item in a list. We may increase the number of TMKs per request in due course and, as such, have built the service to accept a list in the request.

### Our responses
After receiving your list of items, we will match each one against our system and return a list with the same number of items, where an item is the following for cryptographic hashes (where `True` represents a match against content in our archive):

- `hash_value` `(String)`: HASH_DIGEST

- `hash_type` `(String)`:  "MD5" | "SHA256" | "SHA512"

- `result` `(Boolean)`: True | False

- `error` `(String)` | `Null`: An error message or null if successful

and the following for perceptual_hashes:

- `hash_value` `(String)`: HASH_DIGEST

- `hash_type` `(String)`: "TMK" | "PDQ"

- `result` `(Boolean)`: True | False
 
-  `confidence` `(Float)`: 0-1 | [0-1, 0-1]

- `error` `(String | Null)`: An error message or null if successful

### What are confidence scores?
While cryptographic hash verification simply involves checking your hash string for an exact match against our system, perceptual hashes check for similarity between two hashes and their respective pieces of content.

With this in mind, when you request a check on a cryptographic hash, you must include a `confidence` parameter between 0 and 1 where 1 is complete confidence, ie. an exact match. 

We will return `True` in the `result` field if we have a piece of content whose hash matches or exceeds the confidence threshold you provided when compared with your `hash_value`. If we do not have any confidence that matches or exceeds this threshold, we will return `False`.

In addition, if the result is `True`, we will return a `confidence` score that represents just how similar the piece of content on our system was to your piece of content. For PDQ hashes, this score will a decimal between 0 and 1, where 1 is an exact match. For TMK hashes, this score will be a pair of decimals between 0 and 1, where [1,1] is an exact match.

### How do I authenticate to use the service?
Please see our [authentication documentation](https://terrorismanalytics.org/docs/getting-started)

The Hash Verification API provides platforms with an interface to check if any content uploaded by their users has already been identified as terrorist content by the TCAP.

Platforms query the API with the hash strings of their media files from their own systems. In turn, they receive a response indicating if any of those files are, or are likely to be, terrorist content.

Hash Verification (v2.0.0)

### Transforming URLs into Working Variations
At the core of this API's functionality is the innovative process of transforming each incoming URL into up to 13 potential working variations. This is achieved through the application of a sophisticated algorithm that generates these alternatives, which are then hashed using the industry-standard MD5 algorithm.


### Comprehensive URL Hash Retrieval

By leveraging this API, users can effortlessly retrieve the complete collection of MD5 URL hashes from the TCAP database. This comprehensive data set ensures that even the most obscure or hidden links are surfaced, enabling users to maintain a complete and up-to-date record of all terrorist-content-related URLs.

By harnessing the capabilities of the URL Hash List API, you can unlock the full potential of your content, ensuring that no link is left behind and that every piece of information is readily accessible.

In order to use the URL Hash List you will need to be an on-boarded URL Hash List TCAP user. If you're interested in working with us please get in touch.



### Authentication

To obtain a token make a request to the TCAP authentication endpoint with your username and password.

```
POST https://beta.terrorismanalytics.org/token-auth/tcap/
{
  username: YOUR_TCAP_USERNAME,
  password: YOUR_TCAP_PASSWORD,
}
```

### Response

The Authentication endpoint returns the following data on each request:

`token: String` - Token to be used to on each request you make the the API as a `Bearer` token

`user:  User` Your system user information


### URL Hash List 

``` 
GET https://beta.terrorismanalytics.org/core/url-hash-list 

headers = {
  "Authorization": "Bearer {AUTH_TOKEN}"
   ...
}

```

<u>Query Parameters</u>

`from="<YYYY-MM-DD>"`

`to="<YYYY-MM-DD>" `

`limit=<INT> (maximum 1000)`

`offset=<INT>`

`ideology=< "islamist | far-right" > (defaults to all if not provided)`

**Response**

The Hash List endpoint returns a paginated response of URL Hash objects, where each object contains a collection of URL hash digests.

<u>Fields</u>

`"count"` the total number of hashes in the archive when the request was made   

`"next"` endpoint to retrieve the next page 

`"previous"` endpoint to the previous page

`"results"` list of hashes with all variations:

**Example response**

```
200 OK
{
    "count": 1169,
    "next": "https://beta.terrorismanalytics.org/core/url-hash-list?<QUERY PARAMS>",
    "previous": "https://beta.terrorismanalytics.org/core/url-hash-list?<QUERY PARAMS>",
    "results": [
        {
            "hash_digest": "<MD5 HASH DIGEST>",
        },
        ......
    ]
}
```



The URL Hash List API provides a robust solution for retrieving all MD5 URL hashes stored within the TCAP database. This powerful tool is designed to address the challenge of ensuring that no link to a specific piece of content is overlooked. 

URL Hash List API (v1.1.2)

# Key features and use cases

The TCAP Archive is a repository of known terrorist or violent extremist content (TVEC) media files, including images, videos and documents. The TCAP Archive Hash List API allows platforms to ingest the hashes produced from these media files in bulk, so they can use them in their content moderation processes.

The TCAP Archive’s hashes are distinct from any existing TVEC hash lists, as they complement Tech Against Terrorism’s proactive monitoring of terrorist internet usage by its team of open-source intelligence specialists. By leveraging this expertise, alongside a suite of automated monitoring capabilities, the TCAP Archive hash list reflects content created and uploaded over a number of years by a range of violent Islamist and violent far-right terrorist entities.

# Authentication

In order to use any of the Hash List endpoints you will need to be an on-boarded Hash List TCAP user. These endpoints sit inside the main TCAP backend so you will be able to access all with a standard TCAP JWT token.

To obtain a token make a request to the TCAP authentication endpoint with your username and password.

```
POST https://beta.terrorismanalytics.org/token-auth/tcap/
{
  username: YOUR_TCAP_USERNAME,
  password: YOUR_TCAP_PASSWORD,
}
```

### Response
The Authentication endpoint returns the following data on each request:

- `token` : `String` Token to be used to on each request you make the the API as a `Bearer` token

- `user` :  `User` Your system user information


### Hash List by Ideology
```
GET /hash-list/v2/all
```
This Hash List API endpoint retrieves a hash list file filtered by a specified ideology.

### Parameters

- `ideology`:  (Type: String, Required: Yes): Specifies the ideology to filter the results.

```
GET /api/hash-list/:ideology  <'islamist' | ' far-right ' | 'all' >
```

### Response
The Hash List endpoint returns the following data on each request:

- `count` :  `Integer` Total number of hash records available

- `next` : `String` URL of the next page results

- `previous` :  `String` URL of the previous page results (null if first page)

- `checkpoint` :  `String` Timestamp-based checkpoint identifier for synchronization

- `results` : `Array` Array of Hash objects

### Hash Object Fields

- `hash_digest` :  `String` The computed hash value

- `algorithm` :  `"MD5" | "SHA256" | "SHA512" | "PDQ"` The algorithm used to generate the hash

- `ideology` : `'islamist' | 'far-right' | 'all'` Content classification category
 
- `file_type` : `'String' `  Source file format

- `deleted` : `'Boolean' `  If the file has been removed from the system 

- `updated_on` : `'Float' `  Unix timestamp of the last update 

### Pagination
The API implements cursor-based pagination using timestamp and ID pairs. Results can be traversed using the `next` and `previous` URLs provided in the response.

### Query Parameters
- `limit`: Number of results per page (default: 1000)

- `offset`: Starting position for pagination

- `order`: Sort order for results (asc/desc)

- `after`: Cursor value for pagination (format: timestamp,id)

### Implementation Notes
- Multiple hashing algorithms provide redundancy and enhanced detection capabilities

- Checkpoint field enables efficient delta updates for client-side caching

- Each hash entry includes metadata for content categorization and tracking

- Real-time updates reflected through `updated_on` timestamps

- Deleted flag allows for soft deletion while maintaining hash history

### Example Response
```
{
    "count": 19676,
    "next": "http://beta.terrorismanalytics.org/hash-list/v2/all?<params>,
    "previous": null,
    "checkpoint": "1730213563.621023,29152",
    "results": [
        {
            "hash_digest": "baf781254eb82811cdf3fe4751240eb8",
            "algorithm": "MD5",
            "ideology": "Far-right",
            "file_type": "mp4",
            "deleted": false,
            "updated_on": 1730204429.302388,
            "id": 28023
        }
        ...
    ]
}
```

### Best Practices
1. Implement local caching using the checkpoint mechanism

2. Process updates incrementally using the pagination system

3. Consider implementing parallel processing for multiple hash algorithms

4. Store hash values in their original format to maintain precision

5. Monitor the deleted flag for deprecated hash values

### Usage With Metas Threat Exchange
If you want to use the Hash List through [Meta's Threat Exchange](https://developers.facebook.com/docs/threat-exchange/) you can create a collaboration configuration for our API, fetch and compare PDQ Image and MD5 video hashes. 

Step 1 - Install threat exchange

```
$ pip install threatexchange
```

Step 2 - Configure the default credentials

```
$ threatexchange config api tat --credentials '<TCAP_USERNAME>' '<TCAP_PASSWORD>'
```

Step 3 - Set up config

```
$ threatexchange config collab edit tat --create 'TAT'
```

Step 4 - Fetch hashes with verbose logging

```
$ threatexchange -v fetch
```

Step 5 - View dataset

```
$ threatexchange dataset
```

Step 6 - Match a piece of content

```
$ threatexchange match ~/path/to/image.jpg
```

[ThreatExchange docs](https://developers.facebook.com/docs/threat-exchange/)

The Content Hash Fingerprinting API provides robust access to a comprehensive database of content fingerprints designed for advanced content moderation systems. The API leverages multiple hashing algorithms including MD5, SHA256, SHA512, and PDQ to ensure maximum coverage and accuracy in content identification.

Content Hash List API (v2.0.0)

This page contains technical documentation on how to use the Tech Against Terrorism API services. If you are interested in using our services please get in touch through [this form](https://terrorismanalytics.org/about/contact)

DOCUMENTATION

Authenticating for TCAP API Services

TCAP Submissions API (v1.2.0)

Hash Verification (v2.0.0)

URL Hash List API (v1.1.2)

Content Hash List API (v2.0.0)