# Comprehend Media

Comprehend audio or video with Backtracks Media Intelligence to get powerful contextual understanding of what that media is actually about. Backtracks will kick off a comprehension job to understand what the media is about in detail including transcripts with timings of what is actually said, advertising categories that the media maps to, keywords, topics, brand safety scoring and more.

# Comprehend

Comprehend audio and video with media_id. The /comprehend API endpoint is the core endpoint for media intelligence calls.

# Comprehend Request

Initiate a comprehend request to kick-off a job to understand and comprehend the meaning of the media. After submitting the request, the processing and completion status of the job may be verified, and when the job is complete then the results will contain data like a full transcript with timings and the data related to the specified options (i.e. advertising categories).

POST https://api.backtracks.fm/v1/media-intelligence/comprehend HTTP/1.1
Accept: application/json
Authorization: JWT <your backtracks access token>...
Content-Type: application/json

{
   "media_id":"<media_id>",
   "options":{
      "iab_categories":true,
      "keywords":true,
      "entities":true
   },
   "webhook":{
      "url":"https://example.com/abc-123",
      "headers":{
         "Authorization":"Bearer abc123",
         "X-Custom-Header":"456"
      }
   }
}

# Parameters

# Options

The options object represents the types of data requested for the media intelligence comprehend response. If the options object is not specified or any of the options does not receive an explict value then the default value will apply. The transcript flag/option is the only option that has a default value of true if a value for the property is omitted.

# Webhook

The webhook object represents the optional settings for a webhook or callback that is called via HTTP POST on success or error of the request.

# Parameters

# Webhook Payload

Webhooks are sent via HTTP POST to the URL specified in the request. The job_id/id of the comprehend request and the success or error status of the request is sent in the payload. The receiver of the webhook would then call into the API to retreive the data for the request by the job_id/id of the request. Receiving endpoints that error or timeout without returning an HTTP 200 series status code (i.e. 200, 201, 202, etc.) in less than 30 seconds are retried for up to 24 hours.

HTTP/1.1 200 OK
User-Agent: Backtracks/1.0 (+https://backtracks.fm)
Content-Type: application/hal+json; charset=utf-8
Header-Abc: <Header value>

{
    "id": "<id>",
    "status": "success|error"
}

# Comprehend Response

The comprehend job id is returned with the current status of the job. To check on the status of the job and return the result of the job, the API caller would call the self link returned in the response to get the comprehend job status or get the comprehend job result.

HTTP/1.1 202 Accepted
Location: https://api.backtracks.fm/v1/media-intelligence/comprehend
Content-Type: application/hal+json; charset=utf-8

{
    "id": "<id>", 
    "status": "processing",
    "_links": {
        "self": {
            "href": "https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>"
        }
    }
}

# Comprehend Status

When a comprehend job is initiated the system returns the jobs status in the response immediately. The API caller may check on the status of the job and when the job is complete the result will also be returned when the job is complete.

# Get Comprehend Status Request

You may retrieve the status of the comprehend process for a particular comprehend job id with the following API call.

# Parameters

GET https://api.backtracks.fm/v1/media-intelligence/comprehensions/dde0d890-9c13-40de-be68-01c7b1fa9246
Accept: application/json
Authorization: JWT <your backtracks access token>...

# Get Comprehend Status Response

HTTP/1.1 200 OK
Location: https://api.backtracks.fm/v1/media-intelligence/comprehensions/dde0d890-9c13-40de-be68-01c7b1fa9246
Content-Type: application/hal+json; charset=utf-8

{
    "id": "dde0d890-9c13-40de-be68-01c7b1fa9246",
    "status": "processing", 
    "_links": {
        "self": {
            "href": "https://api.backtracks.fm/v1/media-intelligence/comprehensions/dde0d890-9c13-40de-be68-01c7b1fa9246"
        }
    }
}

The primary comprehend job statuses are below. When a comprehend job is complete and successful the status will be success. A comprehend job is called a comprehension.

# Statuses

If the comprehend request by media_id has not started, needs processing, or errored then a response like the following will be returned by the API. The caller may then check on the status of processing by calling the comprehension status endpoints using the id (i.e. job id) returned from the processing response to determine when the data is available.

HTTP/1.1 200 OK
Location: https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>
Content-Type: application/hal+json; charset=utf-8

{
    "id": "<id>", 
    "status": "<queued|processing|success|error>",
    "_links": {
        "self": {
            "href": "https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>"
        }
    }
}

# Success

When a comprehend is complete and successful then a success response is returned with an HTTP 200 status code in the the response. The results of the comprehend job are returned in the response when the job is complete.

HTTP/1.1 200 OK
Location: https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>
Content-Type: application/hal+json; charset=utf-8

{
   "id":"<id>",
   "media_id":"<media_id>",
   "language_code":"en_us",
   "status":"complete",
   "results":[
      {
         "final":true,
         "transcript":{
            "status":"complete",
            "items":[
               {
                  "id":"c0",
                  "start":0.0,
                  "end":11.953,
                  "body":"This is the text of a segment of the transcript. There will be N segments."
               }
            ],
            "_links":{
               "self":{
                  "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>/transcript"
               },
               "srt":{
                  "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>/transcript/srt"
               },
               "vtt":{
                  "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>/transcript/vtt"
               }
            }
         },
         "iab_categories":{
            "status":"complete",
            "items":[
               {
                  "label":"Media Industry",
                  "score":0.3827972114086151,
                  "text":"Backtracks Media Intelligence enables contextual understanding of audio and video at scale. Get automatic transcripts, topic identification, and contextual understanding of media from what is actually said in ads and content.",
                  "tier_1":"Business and Finance",
                  "tier_2":"Industries",
                  "tier_3":"Media Industry",
                  "iab_id":"106"
               },
               {
                  "label":"Advertising Industry",
                  "score":0.3541545867919922,
                  "text":"Backtracks detects the keywords, people, products, dates, companies, organizations, and more, like brand safety categorizations and scoring. It even automatically maps media to advertising categories supported by programmatic ad platforms to enable monetization opportunities. Unlock value through understanding, and learn more at backtracks.fm.",
                  "tier_1":"Business and Finance",
                  "tier_2":"Industries",
                  "tier_3":"Advertising Industry",
                  "iab_id":"91"
               }
            ],
            "summary":{
               "Media Industry":1.907792329788208,
               "Advertising Industry":1.6625179946422577
            }
         },
         "keywords":{
            "status":"complete",
            "items":[
               {
                  "label":"backtracks media",
                  "score":0.26368705417506344
               },
               {
                  "label":"contextual understanding",
                  "score":0.239762634514837
               },
               {
                  "label":"safety categorizations",
                  "score":0.1666656878643069
               }
            ]
         },
         "entities":{
            "status":"complete",
            "items":[
               {
                  "label":"Backtracks Media Intelligence",
                  "type":"PRODUCT"
               },
               {
                  "label":"Backtracks",
                  "type":"ORG"
               },
               {
                  "label":"backtracks.fm",
                  "type":"ORG"
               }
            ]
         }
      }
   ],
   "_links":{
      "self":{
         "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>"
      }
   }
}

# Advertising Categories & Topics

Enable contextual and programmatic advertising in audio and video by mapping what is actually said in media to advertising categories and topics. This granular level of advertising category and topic mapping allows for surfacing contextual data for advertising use cases. For example, in media or media segments about automobiles the most appropriate ads may be automotive ads, ads and content that are both about autombiles may be an appropriate match, etc.

# Advertising Categories Response

In the response from a comprehend request, aka a comprehension, the IAB advertising and content categories are returned if iab_categories is set to true in the options of the comprehend request.

{
   "iab_categories":{
      "status":"complete",
      "items":[
         {
            "label":"Media Industry",
            "score":0.3827972114086151,
            "text":"Get automatic transcripts, topic identification, and contextual understanding of media from what is actually said in ads and content.",
            "tier_1":"Business and Finance",
            "tier_2":"Industries",
            "tier_3":"Media Industry",
            "iab_id":"106"
         }
      ],
      "summary":{
         "Media Industry":1.907792329788208,
         "Advertising Industry":1.6625179946422577
      }
   }
}

# Advertising Categories by Overall Media

Determine the overall advertising categories for a piece of media. For example, the categories for ad creative/media or episode content may be automotive and auto repair. This is represented by the summary property/object of the of the iab_categories.

{
   "iab_categories":{
      "status":"complete",
      "summary":{
         "Automotive":5.570719301700592,
         "Auto Repair":1.907792329788208
      }
   }
}

# Advertising Categories by Segment

In a piece of audio or video media, the topics discussed may change over the duration of the media. For example, the advertising categories associated with one segment of the media are different than the advertising categories associated with another segment where different topics are discussed in the media. This is represented by the items property/object of the of the iab_categories.

{
   "iab_categories":{
      "status":"complete",
      "items":[
         {
            "label":"Media Industry",
            "score":0.3827972114086151,
            "text":"Get automatic transcripts, topic identification, and contextual understanding of media from what is actually said in ads and content.",
            "tier_1":"Business and Finance",
            "tier_2":"Industries",
            "tier_3":"Media Industry",
            "iab_id":"106"
         }
      ]
   }
}

# Brand Safety & Sensitive Content

Automatically determine if media (i.e. ads or content) contains sensitive topics. This is useful for automated ad and content categorization as well as brand safety. Sensitive topics are represented by the summary property/object of the of the iab_categories response data.

{
   "iab_categories":{
      "status":"complete",
      "summary":{
         "Illegal Drugs, Tobacco, eCigarettes, Vaping, Alcohol":1.6625179946422577,
         "Obscenity and Profanity":1.4562954008579254,
         "Online Piracy":1.3775753378868103
      }
   }
}

# Sensitive Topics

# Keywords

Determine exactly what the keywords (or keyphrases) are in the media. This is useful for contextual targeting, search, tagging, and more. When keywords is set to true in the options of the comprehend request, the detected keywords are returned in the success job results response.

# Keywords Response

{
   "keywords":{
      "status":"complete",
      "items":[
         {
            "label":"backtracks media",
            "score":0.26368705417506344
         },
         {
            "label":"contextual understanding",
            "score":0.239762634514837
         },
         {
            "label":"safety categorizations",
            "score":0.1666656878643069
         }
      ]
   }
}

# Topics and Entities

Determine exactly what topics, people, places, etc. are discussed in media. When entities is set to true in the options of the comprehend request, the detected topic and entity types are returned in the success job results response.

{
   "entities":{
      "status":"complete",
      "items":[
         {
            "label":"Backtracks Media Intelligence",
            "type":"PRODUCT"
         },
         {
            "label":"Backtracks",
            "type":"ORG"
         },
         {
            "label":"backtracks.fm",
            "type":"ORG"
         }
      ]
   }
}

# Topic Types

# Accessibility and Captions

Backtracks Media Intelligence creates caption and subtitle formats from generated transcripts. These caption and subtitle formats may be used for accessibility, loading into media players, etc.

# Caption Formats

The Backtracks Media Intelligence API supports converting transcripts to the most utilized caption and subtitle formats with support for srt and vtt.

To convert the transcript to a caption or subtitle format, utilize the _links object and navigate to the key or node for the file format. For exmaple, to access the transcript in the vtt file format find the link in the _links section with the key of vtt.

{
   "transcript":{
      "_links":{
         "self":{
            "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>/transcript"
         },
         "srt":{
            "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>/transcript/srt"
         },
         "vtt":{
            "href":"https://api.backtracks.fm/v1/media-intelligence/comprehensions/<id>/transcript/vtt"
         }
      }
   }
}

For example, the video file below has captions provided in the srt file format.

# Supported Languages

The following 80+ languages are supported for Media Intelligence (in transcription).