Diagnosing discrepancies between analytics systems
Discrepancies are differences in measurement between two systems.
Acceptable discrepancies would be in the 2-5% range at scale.
Identifying the cause of discrepancies is a constructive process, as it means that not as much data is being collected as could be, so this document outlines how to find the cause and resolve it.
- Code placement
- Code not on all templates
- Code placed at different locations in the template i.e code one in the <head> and code two in the <body> or <footer>
- Reporting differences
- Reporting timezones are different, showing different counts
- Metric definitions are different ie Time on page vs Attention Minutes
1. Questions to ask internally
- When the incumbent system was placed, has there been any new updates since then?
- Do you often have problems with incumbent system?
- How are both systems being placed on the page, tag manager, directly in template, through a CMS
2. Check data
- Retrieve impressions by device from both systems.
- Ensure both are on the same time zone.
- Ensure both have data for each day in the time period.
- Note: Exclude the launch day from comparison. This helps remove any go-live mistakes that may have occurred.
- Compare how URLs are tracked in each system.
- Nudge will merge data under one canonical URL. Other systems may fragment these.
- Use the Nudge content stream to see if any unusual url parameters that aren’t being merged.
In most cases, this normally uncovers the cause.
- Does adblocking block either code?
- Nudge isn’t adblocked but may be held within an adblocked container. ie Nudge is rendered through an on page ad placement.
- Code placement on slow loading content may impact performance. This can cause differences of up to 30-40%.
- Use inspector on Google Chrome to find the time to render for both pieces of code.
- Is this an account utilizing domain blocking?
- Are there any geo location restrictions on code loading or dashboard?
- Bot check, other system may not be detecting and removing bots. This can be investigated if all others are exhausted.
Why is Nudge higher? Or lower?
They can be and just as likely the same cause. Because one is higher or lower doesn’t necessarily point to an issue.
Occam’s razor, the most simple explanation is usually the right one
Most causes come down to basic differences between the set up of the two pieces of code.