Enhance Google Analytics with Super Cookies

Google Analytics mechanics are quite different to Omniture Stie Catalyst, for Google Analytics many of the calculations such as pages per visit, first visit, last visit, etc are stored in cookies, they are not calculated on the server side. Additionally the visitor ID is not necessarily the same for a given user, it isn't used to tie information together, instead Google Analytics relies on your cookies telling the truth. The problem is that cookies inherently only ever tell partial truths, the attrition rates are huge, so how can you trust this information? The answer is you can to a point, but be aware of what you're looking at, because its far from perfect. If you want to make it more accurate, then use super cookies. This post is aimed to touch on several of these areas where we think flash can add major value to existing Google Analytics solutions. We've already covered the super cookie technology in a previous post, so we won't dwell on the basics, if you need some more background please read the previous post:

Examples of Super Cookie additions to Google Analytics deployments
  1. Flash based persistent cookies work across multiple domains and multiple browsers - Use super cookies to re-set targeting and other custom variables across your network of domains, in any browser.
  2. Cookie deletion measurements - Find out how often do your existing users delete their standard cookies?
  3. Browser switching measurements - Do your users switch between multiple browsers? How does this affect your analytics?
 
Which browser? Who cares!!!
1. Super persistant cookies
For those wanting to get better accuracy from google analytics, or if you're using the Google Analyitcs custom variables for targeting and reporting, this is for you. 
Persisting the profile - In order to keep the rich information on your users no matter whether they delete cookies or switch browsers, you have two main options:

a) Respawn their past GA cookies before loading GA - All their profile information is associated with their visitor ID. By reseting this to it's original value (stored either prior to cookie deletion, or from a previous browser), their profile remains intact. This method can also help you to keep more realistic figures on unique visitors as long as you can replace the visitor ID prior to sending any requests to Google. Although this gives the smoothest operation, the privacy issues are obvious and must be addressed. 
b) Keep a copy of targeting parameters in a super cookie - If you detect a cookie deletion, resend the the parameters to Google Analytics so they can be re-bind them to the new visitor ID. This is a little more privacy friendly, as you're allowing the user to remove association to a specific ID, but their profile remains. You no longer know who they are, but you still know a little about them to help serve them better.

2. Cookie Deletion Measurements
If you grapple with privacy concerns but are still desperate to know how many of your users delete their cookies, then you can use this method to find out without fear of privacy invasion. This technique is useful for adjusting data inaccuracies caused by cookie deletion. 

Super cookies remain after users delete their standard cookies. Because flash cookies are not currently dealt with by browser settings (Chrome has some functionality), or understood by consumers, they are rarely deleted (assume this will increase in the future). By comparing the super cookie value to the standard cookie value, you can quickly tell if a previous value existed and has since been deleted. The high level logic is found below (note: this over-simplistic and does not allow for browser switching, see section 3!). The following pseudo code would actually be done in JavaScript:

IF standardCookie(a) is not equal to superCookie(a) AND superCookie(a) is not null THEN
{
LOAD GA CODE
Set custom variable to indicate a cookie deletion
SEND GOOGLE REQUEST
} ELSE {
LOAD GA CODE
SEND GOOGLE REQUEST
}

The above logic would enable you to see several things including:

a) The total number of cookie deletions (using the prop or event)
b) Conversion rates of users who have deleted their cookies vs those that haven't (using the custom variable). Note: This is particularly useful for Targeting, where profiling enhances conversion. You can directly measure the uplift of normal users compared to users post cookie deletion.

3. Browser Switching Measurements
Many people now use multiple internet browsers for a variety of reasons, evaluation, different features, old bookmarks and probably most importantly, technical issues. The problem for analysts is that traditional cookies are browser specific, so each browser appears as a different user. Super cookies can quantify this issue. Super cookies provide the capability to keep a cross browser profile that remains even if a user uninstalls a specific browser and switches to a completely new one, but for the purposes of the exercise we are only looking to quantify the issue.

To create this capability the following logic can be used. Again this would be written in JavaScript. 
IF current browser is not equal to superCookie(browser) THEN
{
LOAD GA CODE
set custom variable "browser A > browser B"
SEND GOOGLE REQUEST
set superCookie(browser) = "browser B"
ELSE {
LOAD GA CODE
SEND GOOGLE REQUEST
}

The above logic would enable you to see:

a) Which browsers people are switching from/to. This can help you plan future testing resource allocations, etc.
b) Which pages browser switches are commonly associated with (above logic does not show a direct correlation to a specific page, but you can store the final session page in the super cookie and use that to see if the user has made a browser switch on the same page, which may indicate a browser issue).
c) How many browser switches have occurred (set the variable to be page specific)
d) How many users use multiple browsers (if you keep a common visitor ID across multiple browsers)

Hopefully this article has helped to show you how super cookies can be used to improve your Google Analytics deployment accuracy. For actual code examples, please see our original super cookie post or download the zip file below. For any questions or enquiries, please contact us at insights@datalicious.com

 

 

Use Google Analytics custom variables and simple JavaScript to target site content in real-time

I was reading this on another blog earlier today and thought it was worth passing on. In essence Google Analytics have provided a function to read the custom variables for targeting purposes. Although this is essentially nothing more than being able to read a cookie and use it to segment and target page content, it's nice to be able to use the same variables used by Analytics, as the targeting immediately has context in reports.
 
If you already use the custom variables (index 1-5), you can now use the following function to read the value and switch out content using some simple JavaScript.
 
_getVisitorCustomVar()
 
_getVisitorCustomVar(index)
 
Returns the visitor level custom variable assigned for the specified index.
 
pageTracker._getVisitorCustomVar(1); 
 
Parameters
 
Int index The index of the visitor level custom variable.
 
Returns
 
String The value of the visitor level custom variable. Returns undefined if unable to retrieve the variable for the specified index.
 
Read the original blog post from Michael Whitaker here or check out the code reference in Google's help section if you want to find out more.

Extending Omniture Analytics with Flash based Super Cookies

Adobe now owns Omniture, so like everyone else, we're expecting some merging of their respective technologies in the coming few years. Flash based super cookies are an ideal compliment to Site Catalyst and Test & Target, but despite the ease of implementation and widespread usage of both technologies, there is minimal current overlap. 
Given we are always trying to squeeze more value from analytics, we started playing with flash based super cookies and once we got it working, we were amazed at the possibilities. This post is aimed to touch on several of these areas where we think flash can add major value to existing Omniture solutions. We've already covered the super cookie technology in a previous post, so we won't dwell on the basics, if you need some more background please read the previous post:

Examples of Super Cookie additions to Omniture deployments
  1. Flash based persistent cookies work across multiple domains and multiple browsers - Standard cookie attrition reduces Test and Target profiling effectiveness and consequently reduces the potential uplift the technology can provide.
  2. Cookie deletion measurements - How often do your existing users delete their standard cookies?
  3. Browser switching measurements - Do your users switch between multiple browsers? How does this affect your analytics?

1. Super persistant cookies
For those people already using Omniture Site Catalyst's s.getAndPersistValue plugin, this is can be thought of as a substantially improved version, it doesn't matter if the user switches browser, the cookie exists in all browsers. It doesn't matter if the user clears their cache, deletes their cookies, it still persists. For analytics heavily dependent on a client side value persisting, this is invaluable, it directly translates to significantly more accurate analytics
For those using Test and Target the value of the super cookie is even more pronounced. Targeting parameters like product and category affinity often take many page views and multiple sessions before profiling parameters can be accurately determined. When standard cookies are deleted, all that rich information is lost. While people think they've done the right thing deleting their cookies to protect their privacy, they may also be perplexed at why your site suddenly starts showing them products and services they have absolutely no interest in. There is very little awareness that the cookie deletion has actually devalued the service provided to them. 

Persisting the profile - In order to keep the rich information on your users no matter whether they delete cookies or switch browsers, you have two main options:

a) Respawn their visitor ID - All their profile information is associated with their visitor ID. By reseting this to it's original value (stored either prior to cookie deletion, or from a previous browser), their profile remains intact. This method can also help you to keep more realistic figures on unique visitors as long as you can replace the visitor ID prior to sending any requests to Omniture. Although this gives the smoothest operation, the privacy issues are obvious and must be addressed. 
b) Keep a copy of targeting parameters in a super cookie - If you detect a cookie deletion, resend the the parameters to Omniture so they can be re-bind them to the new visitor ID. This is a little more privacy friendly, as you're allowing the user to remove association to a specific ID, but their profile remains. You no longer know who they are, but you still know a little about them to help serve them better.

The image above is an example of respawning an identifier which would have long since expired when using traditional cookies. The flow shows how the customer is already known and because a lead situation has been identified, he will be called by a sales rep. 

2. Cookie Deletion Measurements
If you grapple with privacy concerns but are still desperate to know how many of your users delete their cookies, then you can use this method to find out without fear of privacy invasion. This technique is useful for adjusting data inaccuracies caused by cookie deletion. 

Super cookies remain after users delete their standard cookies. Because flash cookies are not currently dealt with by browser settings (Chrome has some functionality), or understood by consumers, they are rarely deleted (assume this will increase in the future). By comparing the super cookie value to the standard cookie value, you can quickly tell if a previous value existed and has since been deleted. The high level logic is found below, this would be done in JavaScript:

IF standardCookie(a) is not equal to superCookie(a) AND superCookie(a) is not null THEN
{
s.propXX = "cookie deletion"
s.eVarXX = "cookie deletion"
s.events = "eventXX"
SEND OMNITURE REQUEST
}

The above logic would enable you to see several things including:

a) The total number of cookie deletions (using the prop or event)
b) Conversion rates of users who have deleted their cookies vs those that haven't (using the eVar). Note: This is particularly useful for Test and Target, where profiling enhances conversion.

You can directly measure the uplift of normal users compared to users post cookie deletion.

3. Browser Switching Measurements
Many people now use multiple internet browsers for a variety of reasons, evaluation, different features, old bookmarks and probably most importantly, technical issues. The problem for analysts is that traditional cookies are browser specific, so each browser appears as a different user. Super cookies can quantify this issue.

Sadly compatibility issues affect all browsers, although i shudder to think of the man hours wasted catering for IE6 issues. For example, a while back I emailed Citibank to let them know their site was completely crippled in Firefox, which was my browser of choice at the time (incidentally they never responded, i assume they were too busy issuing CDO's or something...). I didn't have Internet Explorer or any other browser, so i had to go and install it just to access Citibank. I continued to check back in Firefox from time to time waiting for the issue to be fixed and eventually it worked, but now i use Chrome... My experience with Citibank is fairly common, i've used 3 browsers in past 12 months as my primary browser, one switch was by choice and one was forced by technical issues, if Citibank didn't require me to identify myself by logging in, they would think i was 3 different users. 

So what does all this mean for super cookies? In the above case, Citibank may have assumed there was no need to support Firefox because none of their customers used it, but in reality, they couldn't use it because it didn't work. What if they had a way to see that i had visited several pages in Firefox and on a particular page i switched browsers. This would provide them with the critical point where a technical issue was occurring. 

Looking beyond the Citibank example, super cookies provide the capability to keep a cross browser profile that remains even if a user uninstalls a specific browser and switches to a completely new one, but for the purposes of the exercise we are only looking to quantify the issue.

To create this capability the following pseudo code logic can be used. Again this would be written in JavaScript. 
IF current browser is not equal to superCookie(browser) THEN
{
s.propXX = "old browser > new browser"
s.eVarXX = "old browser > new browser"
s.events = "eventXX"
SEND OMNITURE REQUEST
set superCookie(browser) = current browser
}
 

The above logic would enable you to see:
a) Which browsers people are switching from/to (s.propXX). This can help you plan future testing resource allocations, etc.
b) Which pages browser switches are commonly associated with (above logic does not show a direct correlation to a specific page, but you can store the final session page in the super cookie and use that to see if the user has made a browser switch on the same page, which may indicate a browser issue).
c) How many browser switches have occurred (s.eventXX)
d) How many users use multiple browsers (if you keep a common visitor ID across multiple browsers)

Hopefully this article has helped to show you how super cookies can be used to improve your Omniture Analytics deployment accuracy. For actual code examples, please see our original super cookie post or download the zip file below. For any questions or enquiries, please contact us at insights@datalicious.com

 

Click here to download:
Datalicious_Super_Cookie.zip (9 KB)

E*Trade: Omniture SiteCatalyst implementation review including media attribution and optimisation

Challenge

To quantify the impact of offline and online media spend on new customer acquisition, along with the construction of a reporting and analytics framework to continually optimise media budget allocation using a better understanding of customer vs. non-customer navigational paths. 

Solution

Major review and upgrade of the existing Omniture SiteCatalyst deployment to build a data collection foundation that could properly address the core business KPIs. A controlled display exposure test was used to quantify the incremental conversion uplift from online display advertising and to determine the real media cost per acquisition. In addition, a website entry survey was used to further analyse the impact of existing brand equity and offline media activity on online channels, specifically navigational search and direct to site visits. It was particularly important to look at the differences between customers and non-customers.

Results

E*Trade now has the ability to report on and de-duplicate conversions across all paid and non-paid media channels as well as to clearly differentiate between customers and non-customers. The marketing team can now not only optimise the company's media mix for maximum acquisition efficiency but also use the data to develop a smoother customer experience and media attribution model. The developed reporting and analytics framework provides the client with the ability to analyse a range of core KPI's on an ongoing basis, putting the business in an excellent strategic position moving forward.

Testimonials

"E*Trade Australia was originally looking for a provider who could help us optimise our media spend and Datalicious brought with them best practice analytics to demonstrate the true value of our marketing dollars. Since then, they have become a critical business partner in their support of our Omniture analytics implementation. They've provided great insights which have driven key business decisions." Trang Young, Senior Internet Marketing Manager, E*Trade Australia

 

Super Cookies example code using Flash local storage

What are Super Cookies?
Super cookies are a relatively new method of storing user information locally for usage in web applications. Users will be familiar with standard web cookies, which are used to store anything from session ID's through to personal information and usage information. Their downsides are many, mainly that they get deleted all the time, or are simply rejected based on the users privacy settings. Along with straight up rejection, cookies are also browser specific. If you're like me and switch from chrome to firefox to safari on a regular basis, then analysts are out of luck, each browser appears in reports as a different user. Just to compound those issues a little more, cookies are domain specific. With the exception of third party cookies (set on another domain), which are largely rejected these days, all cookies are stored under the domain the visitor is visiting. This is also the only domain that can access those cookies, which means sites or networks spanning multiple domains have major issues trying to pass information as users hop domains. 
Enter the Super Cookie...
Imagine you can set a cookie that is available across multiple domains and multiple browsers, with a high acceptance rate and a dramatically lower deletion rate. You can! Here are some of the advantages:
  • Flash cookies use the Flash local storage object (LSO), which is common to all browsers. 
  • Flash cookies are set on the domain the flash file was served from, so by serving from a common location, the cookies are available to multiple domains.
  • Flash has a 99% penetration rate (according to them - http://www.adobe.com/products/player_census/flashplayer/)
  • The cookie size limitations are far less restrictive than standard cookies.
  • Flash content is largely not controlled by browser privacy settings (yet). So the deletion rates are minimal compared to standard cookies
  • Flash can be used to back up standard cookies so they can be regenerated 

How to install?
We've created a very simple integration that essentially provides the ability to read and write cookies with just a few lines of code to be cut and paste, see below. For full details, please see the attached zip file with code and fully documented examples.
 
Essentially there are 3 steps:
1. Add the js and swf files to your server (see "Datalicious Super Cookie.zip" file)
2. Cut and paste the following code on your page
 
<script type="text/javascript" src="supercookie.js"></script>
<script type="text/javascript">
function dtFlashCookieLoaded(){
// Function triggers when the flash loading is complete. At this point you can read and write the super cookies. 
}
</script>
 
3. Once you have added the files and put the code on page, cookies can be read/written using two very simple javascript functions, as follows:
setSuperCookie(name,value);
getSuperCookie(name);
 
The dtFlashCookieLoaded() function is executed once the flash has loaded. If you have code depending on the flash read/write, you can put it inside this function or alternatively you can use this to set a flag to indicate that the flash object can now be accessed.
 
Custom Implementations
Datalicious provides consulting on web analytics, business intelligence and strategy. Feel free to contact us with any enquiries at insights@datalicious.com
 
 
Privacy controls
Lots of people are not happy about super cookies, the privacy concerns are obvious. In some ways i can support this notion, but as a web analyst i also see the benefits passed on to consumers by the better understanding of customers, and they are huge. We encourage businesses to be open and transparent about privacy. If you are concerned about privacy, please see the following Wikipedia article http://en.wikipedia.org/wiki/Local_Shared_Object
 

 

Click here to download:
Datalicious_Super_Cookie.zip (9 KB)

Big blow for web analytics: Google search is going secure and along the way hiding all referrer data

Google search is currently in the process of beta testing a secure search, check it out yourself at https://www.google.com.
 
Even though this might be commendable from a privacy perspective, the move could potentially have devastating effects for all search marketers and online analysts. Turning the normal Google search results into secure pages will essentially hide all usage information including the referrer string which contains data on the used search term. So if you would like to know what keywords are driving your business, you're out of luck. For all secure searches, standard analytics software packages will no longer be able to see the what domain referred the visitor (i.e. search engine) and what search term was used (see screen shots below).
 
It's rumoured that China may be part of the motivation for the switch as this will prevent filtering of certain keyword searches as well as the ability to packet sniff the usage of Google. The switch hasn't rolled out in Hong Kong yet, but you can safely assume it won't be too far away!
 
For more information check out the following Google and Wired articles.

Google Webmaster Tools gets more transparent, now shows impressions and clicks by result position

In case you didn't catch it, Google Webmaster Tools recently deployed a VERY useful new feature. You can now break your sites search traffic down by the query keywords and also your position in the results. These new stats show how many times you appeared in the results for a given search (impressions) and also how you performed (clicks and CTR). The data also breaks down which pages the users were sent to on your site.

You wouldn't have to be Einstein to know Google is definitely using this information to adjust search result rankings. It's bringing more of an adwords style performance metric into the search algorithm. If people love clicking your link, then it's probably a good result for that search. This type of adjustment obviously helps to weed out crap results, but if it's a well formed result listing to a bad site it may not matter too much (Note: that's likely where Chrome and/or Google Analytics usage data are integrated)

Below shows how the Datalicious web site is performing for the search term "osama bin laden" (Note: We were surprised to see we somehow get traffic off this term too!). For a given query you can see the traffic, and the click through rates for each position. This is not a fantastic example as the click through is too low to compare, but for high traffic terms you will notice wild variations in the click through percentage of each position in the search results. 


Tynt: Leveraging the copying and pasting of your website content for search optimisation

Tynt is an interesting piece of technology that deals with people copying text from your web site. The software essentially adds a backlink to the copied text that is also trackable. The result is that when people copy the text, more often that not they will also paste your link.
 
The benefits of this technology are pretty obvious, more back links which increases traffic and aids SEO, as well as additional analytics to help you better understand your site content.
 
The company has a patent pending on the technology and it is apparently already used by thousands of sites. It is free and the total javascript payload is only approximately 12k. Worth a try, particularly for content rich sites ... 
 
For more details, see http://www.tynt.com/
 

Implement Google Analytics server side without JavaScript to track RSS and XML based content

If you've ever been frustrated at the need for developers to add javascript tracking code, or you simply want to track pages like RSS and mobile pages where javascript is not allowed, then the following solution is for you.

We've done several server side solutions like this, such as the integration of phone calls into Google Analytics directly from the phone routing server (stay tuned for more!). I was reading through some articles researching and i came across an elegant post that i thought i should share:

http://www.vdgraaf.info/google-analytics-without-javascript.html

The PHP code to fake the image request for Google is as follows (also found at http://www.vdgraaf.info/wp-content/uploads/tracker.txt), although a similar technique can be used for Omniture or other platforms. This post is pretty old, but the code should still be fine, the principle is definitely still ok! 

Oh and this is also useful if you want to add other information to your Analytics, but you want to keep it from prying eyes! If you find this post useful, you may also want to check out one of our posts on network packet sniffing, which is becoming a serious alternative to javascript page tagging. See http://blog.datalicious.com/atomic-labs-pion-implementing-omniture-withou 

 

For more help with these types of integrations, contact us at insights@datalicious.com

<?php
$var_utmac='UA-000000-1'; //enter the new urchin code
$var_utmhn='yourdomain.com'; //enter your domain
$var_utmn=rand(1000000000,9999999999); //random request number
$var_cookie=rand(10000000,99999999); //random cookie number
$var_random=rand(1000000000,2147483647); //number under 2147483647
$var_today=time(); //today
$var_referer=$_SERVER['HTTP_REFERER']; //referer url

$var_uservar='-'; //enter your own user defined variable
$var_utmp='tracker/'.$_GET['url'].'.'.$_GET['filetype']; //this example adds a fake file request to the (fake) tracker directory (the image/pdf filename).

$urchinUrl='http://www.google-analytics.com/__utm.gif?utmwv=1&utmn='.$var_utmn.'&utmsr=-&utmsc=-&utmul=-&utmje=0&utmfl=-&utmdt=-&utmhn='.$var_utmhn.'&utmr='.$var_referer.'&utmp='.$var_utmp.'&utmac='.$var_utmac.'&utmcc=__utma%3D'.$var_cookie.'.'.$var_random.'.'.$var_today.'.'.$var_today.'.'.$var_today.'.2%3B%2B__utmb%3D'.$var_cookie.'%3B%2B__utmc%3D'.$var_cookie.'%3B%2B__utmz%3D'.$var_cookie.'.'.$var_today.'.2.2.utmccn%3D(direct)%7Cutmcsr%3D(direct)%7Cutmcmd%3D(none)%3B%2B__utmv%3D'.$var_cookie.'.'.$var_uservar.'%3B';
 
$handle = fopen ($urchinUrl, "r");
$test = fgets($handle);
fclose($handle);

switch ($_GET['filetype']){
	case 'jpg':
	header('Content-Type: image/jpeg');
	break;
	case 'gif':
	header('Content-type: image/gif');
	break;
	case 'pdf':
	header('Content-type: application/pdf');
	break;
	// add your own content types where needed
}

$imageurl = fopen ('http://www.yourdomain.com/'.$_GET['url'].'.'.$_GET['filetype'], "r"); //this is where the real file should be located
while (!feof ($imageurl)) {
	$image = fgets($imageurl, 4096);
	echo $image;
}
fclose($imageurl);
?>

 

Open source column based data warehouses for affordable high performance data mining and storage

What analyst or marketer hasn't at some stage been frustrated by the excruciatingly slow data crunching speeds of flat Excel files or standard transactional row based databases such as MySQL when trying to analyse large sets of data? Some of you might have been able to afford professional data warehouse solutions and the necessary hardware to increase performance but the majority probably not.

Well, the arrival of open source column based data warehouse solutions is going to change all that and make professional high speed data mining affordable for small to medium size businesses. All analysts and marketers are now able to build enterprise grade data warehousing platforms in order to develop single customer views or to create powerful business intelligence platforms.

Check out the articles and graphs below to find out more about column based databases in general, how they work and how much faster they are compared to even enterprise grade systems on high performance hardware. 

Email us at insights@datalicious.com if you would like to explore how Datalicious could help your company to set-up a single customer view or business intelligence data warehouse.

MySQL: Why should I check out a MySQL-based column database?

Wikipedia: Column orientated database management systems

   
Click here to download:
column_based_data_warehouse_an.zip (217 KB)