Analyze Models

Overview

"Predictive Models" are software entities that combine "data mining" and statistic modeling techniques to predict whether the display of a particular ad will have the desired result. They are more sophisticated than "bidding rules", which typically include just a few considerations. Models take into account a great many factors, including, for example, what has been shown on the device already, the amount of success the ad has had in general, the CPA goal, recency of the last ad shown, and many others. Armed with all of this information, models determine whether or not to bid on a particular ad placement opportunity and if it does decide to bid, the bid amount.

In addition to the considerations that models originally take into account, it's important to realize that as the campaign continues, the model collects more and more data that permits it to learn more and more and thus make better and better bidding decisions.

For more on model concepts, see De-Mystifying Models (TBD - NEED UPDATED LINK) at the main Zeta site.

Zeta has invented models in various categories such as Conversion, Click, Viewability, Uplift or Age/Gender. There are multiple types and models in each category. For example, model types in the Conversion category include those intended for new campaigns, for difficult conversion situations, for click-through campaigns, etc.

The platform presents models at the Line Item level.

Opening the Models Report

  1. With the campaign open, click the caret next to Manage Campaign (atop the page), and then click Campaign Analytics. The Report Builder appears. From this report, you can access all other Insights reports.

  2. Click the caret next to Report Builder, and select Models > Analyze Models from the drop-down menu.

      3. The Analyze Models screen includes the following tabs. 

  • Models – see which model the campaign is using and a summary of how it has performed

  • Performance – study campaign model performance graphically, with the ability to dive down to the Line Item level

  • Calibration – study model calibration, i.e. how well has it performed compared to what it expected, with the ability to dive down to the Line Item level

  • AUC History – Area Under the Curve study of the model

  • ROC Chart – Receiver Operating Characteristic results, a study of how well the model differentiates the best from the worst impression opportunities

Choose any tab to access the corresponding screen.

Models

This screen does not present any data currently.

Performance

The Performance screen provides information on how well the campaign and each of its model has performed.

Choose the calendar icon to set the date range to analyze. The platform displays a popup. Use it to choose the dates you want and then Apply.

Graph

By default, the histogram displays Conversion counts (you may change the topic using Metrics at right) against time. The red line indicates the entire campaign while each bar reflects the performance of an individual model.

Moving Average

A moving average analyzes data points by creating a series of averages of different subsets of the full data set. Given a series of numbers and a fixed subset size, the first element of the moving average is obtained by taking the average of the initial fixed subset of the number series. Then the subset is modified by "shifting forward"; that is, excluding the first number of the series and including the next number following the original subset in the series. This creates a new subset of numbers, which is averaged. This process is repeated over the entire data series. The plotline connecting all the (fixed) averages is the moving average.

A moving average enables you to smooth out short-term fluctuations and highlight longer-term trends. Drag the Moving average slider right or left to control the subsets of data that the graph includes.

Fix y-axis

Check this box to fix the height of the graph's vertical axis when you change Metrics (see next), permitting you to more easily compare various metrics. Otherwise, the platform customizes the y-axis to best fit the data for each particular metric.

Metrics

If you wish to study metrics other than conversions, choose this item. 

Choose one of the metrics to alter what the graph displays:

  • CTR: Click-Through Rate

  • CVR: Conversion Rate

  • CPA: Cost Per Acquisition

  • CPC: Cost Per Click

Legends

Hover over the name of any model or that of the campaign to highlight its data in the graph. Uncheck an item to prevent the platform from highlighting it.

View Control

The control at the bottom controls the graph view. Grab the vertical border at either end of this control and drag it left or right. The platform excludes the dragged over data from the graph and widens the view of the remaining data so that you can study it in depth.

Calibration

Use this screen to study how well the campaign's models are calibrated, that is, answering the question "did the model behave as expected?"

Graph

The screen displays calibration for each model as well as the entire campaign, calculating calibration as follows:

  • (Predicted Actions – Observed Actions) ÷ (1 + the maximum of Predicted or Observed Actions)

  • Therefore a score of 0 indicates the best calibration.

  • Hover over a particular data point to display its actual numeric value. 

Moving Average 

See above for an explanation of this control.

Fix y-axis

Check this box to fix the height of the graph's vertical axis when you change Metrics (see next), permitting you to more easily compare various metrics. Otherwise, the platform customizes the y-axis to best fit the data for each particular metric.

Legends

Hover over the name of any model or that of the campaign to highlight its data in the graph. Uncheck an item to prevent the platform from highlighting it.

View Control

The control at the bottom controls the graph view. Grab the vertical border at either end of this control and drag it left or right. The platform excludes the dragged over data from the graph and widens the view of the remaining data so that you can study it in depth.

AUC History

Analyze the Area Under the Curve for the model Receiver Operating Characteristic (ROC). The AUC  is a measure of how well the model is ordering the impressions by score. Choose the calendar icon to set the date range to analyze. 

Graph

  • The higher the value, the better the model's performance.

  • Hover over a particular data point to display its actual numeric value.

  • The value of this chart is less, however, in the actual value at any given point, but in seeing the recent trend, say the last thirty days. If the AUC is lately degrading it implies the model is no longer finding high-value conversions (or whatever metric the campaign uses).

Moving Average

See above for an explanation of this control.

Fix y-axis

Check this box to fix the height of the graph's vertical axis when you change Metrics (see next), permitting you to more easily compare various metrics. Otherwise, the platform customizes the y-axis to best fit the data for each particular metric.

Legends

Hover over the name of any model or that of the campaign to highlight its data in the graph. Uncheck an item to prevent the platform from highlighting it.

View Control

The control at the bottom controls the graph view. Grab the vertical border at either end of this control and drag it left or right. The platform excludes the dragged over data from the graph and widens the view of the remaining data so that you can study it in depth.

ROC Chart

This screen does not present any data currently.