Business Intelligence: Real World Insights

Business Intelligence: Real World Insights

Real-world Insights from Mining Retail E-Commerce Data Ronny Kohavi, Ph.D. Vice President, Business Intelligence Blue Martini Software San Mateo, CA http://www.kohavi.com http://www.bluemartini.com/ bi May 22, 2003 Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 1 Goals Give you a feel for what e-commerce data looks like Show interesting insights with fun teasers from Blue Martini customers data Show things that worked well for us,

including architecture and powerful visualizations Next week: share more detailed data mining lessons and challenges (Rajesh Parekh) Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 2 Agenda Overview of architecture Usability Web site traffic Timeout

Searches, referrers Micro-conversions and utilizing real-estate E-mail campaigns Multi-channel analysis Cross-sells / Associations Classification Summary Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA

3 The Vision in 1998 In July of 1998, I gave an invited talk at ICML titled Crossing the Chasm: From Academic Machine Learning to Commercial Data Mining http://robotics.stanford.edu/users/ronnyk/chasm.pdf Most talks have one key slide (some have zero ) The key slide was the following slide, which guided the design of the data mining architecture at Blue Martini software Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 4 Key Slide in Crossing the Chasm Our CEO did this once

before Vertical: e-commerce retail - Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 5 Integrated Architecture Stage Data Business Data Definition Customer Interactions marketplace (Web, campaigns, Call Center, Wireless, POS)

(Enterprise Desktop, Remote Desktop) Deploy Results Build Data Warehouse (DSSGen) Analysis (Reporting, Analytics, Visualizations, OLAP) Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 6 Advantages of Architecture It is well documented that 80% of the time spent in knowledge discovery is spent on data preparation Our architecture shares enough meta data and there is enough

domain knowledge to cut that dramatically Clickstreams Store from the application server layer to the DB (no need to load from flat files on multiple web servers, conflate, and sessionize) Collect additional information (screen resolution, local time) Tie all activities (registrations, orders) to sessions Log high level Business Events, including cart activities, search information, form errors More information in Integrating E-Commerce and Data Mining: Architecture and Challenges, ICDM 01 Available at http://robotics.stanford.edu/users/ronnyk Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 7 Usability Form Errors This was the Bluefly home page Looking at form errors logged by our architecture, we saw thousands of errors every day on this page

Any guesses? Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 8 Improved Home Page This is the new Bluefly home page Search box added E-mail box clearly marked as email As with many insights, hindsight is 20/20 The hard part is collecting the right information and reporting on it Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 9 Bot Detection Bots

are automated programs, sometimes called crawlers/robots Examples: search engines, shopping bots, performance monitors Significant traffic may be generated by bots Can you guess what percentage of sessions are generated by bots? 23% at MEC (outdoor gear) 40% at Debenhams Without bot removal, your metrics will be inaccurate We find about 150 different bot families on most sites. Very challenging problem! Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 10

Example: Web Traffic Weekends Sept-11 Note significant drop in human traffic, not bot traffic Internal Performance bot Registration at Search Engine sites Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 11 Heat Maps for Day-of-Week (Same Data) Use color to show an additional dimension Green is low traffic Yellow is medium traffic Red is high traffic

The power of visualizations Weekends are very slow Friday is slow Patterns Sept 11 in green Reduced traffic after Sept 11 (yellow above Sept 11) Sept 3 Labor day in green Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 12 Browsing hours Traffic by hour (server time) Lines show two consecutive weeks

What do you think it looks like? How stable is it across domains/geographies? EST CST Tokyo Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA GMT 13 Drill-Down to Hour Same heat map idea applies to hourly patterns In

this case hourly traffic to a web site Note Sept 11 effect and its effect for rest of week Site down at critical hour Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 14 Teaser Here is a similar heatmap Interestingly, the white square (no traffic) appeared on many sites

But not in Phoenix, AZ servers Why? Site down? Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 15 Teaser We found that people purchase hours after visiting the site 9 8 7 Percent 6 5 4 3

2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Hour Percent - Clicks Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Percent Order lines 16 Session Timeout Catledge and Pitkow in a well referenced paper determined that the optimal session timeout for analysis should be 25.5 minutes How many visitors at Debenhams Added product to shopping cart Waited over 25.5 minutes Came back to the site in the next 3 hours? 5% (right axis)

Thats a lot of lost shopping carts Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 17 Searches Architecture records every search and the number of results Top searched keywords (percent of searches) Empty search string (3.9%) returns over 160 results Recommendation: GPS (1.2%) - Do not allow empty search sunglasses (0.8%) - Create custom pages for often searched keywords

Top failed keywords in the product category (percent of failed searches) gift certificate(s) (0.98%) (already implemented since study) Recommendation: arcteryx (0.44%) - Parse search string to remove special characters bear spray (0.44%) pedometer (0.37%) stroller(s) (0.36%) - Build extensive thesaurus - Consider carrying products Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 18 Synonyms At Publix, an online grocer in the southeast, Bath Tissue was among the top selling assortments

Top failed search? Toilet Paper Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 19 Search Effectiveness at MEC Customers that search are worth two times as much as customers that do not search Failed searches hurt sales Visit 10% 90% No Search Search

(64% successful) Avg sale per visit: $X Avg sale per visit: 2.2X 70% 30% Last Search Failed Last Search Succeeded Avg sale per visit: 2.8X Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Avg sale per visit: 0.9X 20 Referrers at Debenhams Top Referrers MSN (including search and shopping) Average purchase per visit = X

Google Average purchase per visit = 1.8X AOL search Average purchase per visit = 4.8X Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 21 Micro-Conversion Rates at Debenhams Understand abandonment and conversions Not just visitor to purchaser, but also the micro-conversions Shopping Cart Abandonment 62% =55% + 45% * 17%

Excellent opportunity to identify problematic steps in processes and improve Also a good way to identify abandoned products, send targeted e-mails if those products are on sale Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 2.0% 25% 7.7% 2.3% 6% 45%

83% 17% 55% 22 Page Effectiveness Percentage of visits clicking on different links 14% 3% 2% 8% 2% 13% 9% 0.6% Top Menu 6% 3% 2% 2%

18% of visits exit at the welcome page Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 0.3% 2% Any product link 7% 23 Top Links followed from the Welcome Page: Revenue per session associated with visits 1.4X X 2.3X 2.3X 1.3X 4.2X 5X 1.4X Top Menu 0.2X

10X 10.2X 1.2X 1.7X 3.3X Note how effective physical catalog item #s are Product Links 2.1X Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 24 Teaser - High Conversion Rates Product Conversion Rate is the ratio of product purchases to product views High can conversion rates be over 100% Conversion rates are high because Call Center (orders but no views)

Automatic reordering (send me the medicine every month) Bundles (you view X, you get Y for free) Wizard (at Virgin Wines, they mix you a case; most people dont even look at the details) Quantities over 1 (question of exact definition of conversion) Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 25 Teaser - Privacy 92% of Americans are concerned (67% very concerned) about the misuse of their personal information on the Internet. - FTC Report, May 2000 86% of executives dont know how many customers view their privacy policies. - Forrester Report, November 2000 Q: What percentage of visitors read the privacy statement?

A: Less than 0.3% Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 26 Direct Mail Campaigns (Why Spam) Assumptions: Response rate: 3% (This is 0.6% for credit-card solicitations now, but were going to send a wonderful offer for our Widget and get 3% response) Average revenue per response: $100 Profit margin: 20% (after all costs, including handling returns, shipping, etc.) To breakeven, how much should the offer cost per person? Think about: creative design costs, letter, brochure, outer envelope, reply envelope, stamp, per-person cost when purchasing list Cost should be less than 60 cents! 3%*$100*20% = $0.60 Obviously, its not an easy business Thats why e-mail spam are so cost effective

Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 27 Campaign Analysis - Debenhams Analyze the effectiveness of campaigns Campaign Campaign 1 Campaign 2 Campaign 3 Emails Sent Opens Clickthroughs 100% 22% 9.3%

(4.8p/email) (22.3p/email) (52p/email) 100% 11% 3% (0.5p/email) (4.8p/email) (17.9p/email) 100% 22% 5.3% (0.8p/email) (3.6p/email) (15.3p/email)

Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Orders 0.07% 0.01% 0.01% 28 Multi Channel Analysis Multichannel customers spend 72% more per year than single channel customers -- State of Retailing Online, shop.org If we define a multi channel customer to have shopped on the web and at a store How much more do multi channel customers spend at over single channel customers? More

than twice as much for customers with two or more purchases (you cant be multi-channel if you havent shopped twice). Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 29 Channels by Num Purchases The following graph shows that for each known number of purchases, the web-channel-only customer is better 2000 Wrong! 1800 Customer Average Spending Therefore, our intuition tells us that the web channel is the best channel, right? 1600 1400

1200 1000 800 600 400 200 0 1 2 3 4 5 >5 Number of purchases Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Multi- channel Web- channel only 30

Bug? Multi-channel customers have higher total spending This is an example of Simpsons paradox Customer Spending 400 300 200 100 0 Multi-channel Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Web-channel only 31 Simpsons Paradox

A woman sues Stanford for sex bias She shows that the school admits 70% of males but only 56% of females Stanford agrees with these percentages Shows that in every department they accept a higher percentage of females than males What is amazing is that this can happen What is more amazing is that it happened in practice Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 32

Subtle Difference in Conversation Alice to Bob: Im applying to Stanford next year Bob to Alice: Sorry to hear that; I know theyre accepting more males than females VS Alice to Bob: Im applying for department X at Stanford next year Bob to Alice: Lucky you, I know theyre accepting more females than males in department X And it doesnt matter what X is! Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 33 Here is a Simplified Version 30 customers

Total spending 300 customers Average Average 100 customers Blue web channel Green multi channel with web 200 customers 2 >5 Purchases The web channel dominates the multi-channel with web in both 2-purchases and >5 purchases Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 34 Product Affinities

Which products sell well together Discovered using the association algorithm For closing the loop, associations can be used to make cross-sell recommendations at the website Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 35 Product Affinities at MEC Product Orbit Sleeping Pad Bambini Tights Childrens Silk Crew Womens

Cascade Entrant Overmitts Association Orbit Stuff Sack Bambini Crewneck Sweater Childrens Silk Long Johns Womens Polartec 300 Double Mitts Lift Confidence 222 195 37%

Website Recommended Products Cygnet Sleeping Bag Aladdin 2 Backpack 52% Yeti Crew Neck Pullover Childrens 304 Beneficial Ts Organic Long Sleeve T-Shirt Kids 73% Micro Check Vee Sweater 51 Primus Stove Volant Pants

Composite Jacket 48% Volant Pants Windstopper Alpine Hat Tremblant 575 Vest Womens Minimum support for the associations is 80 customers Confidence: 37% of people who purchased Orbit Sleeping Pad also purchased Orbit Stuff Sack Lift: People who purchased Orbit Sleeping Pad were 222 times more likely to purchase the Orbit Stuff Sack compared to the general population Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 36

Product Affinities at Debenhams Product Fully Reversible Mats Association Egyptian Cotton Towels Lift 456 Confidence 41% Website Recommended Products J Jasper Towels

Confidence 1.4% White Cotton T-Shirt Bra Plunge T-Shirt Bra 246 25% Black embroidered underwired bra Confidence 1% Minimum support for the associations is 50 customers Confidence: 41% of people who purchased Fully Reversible Mats also purchased Egyptian Cotton Towels

Lift: People who purchased Fully Reversible Mats were 456 times more likely to purchase the Egyptian Cotton Towels compared to the general population Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 37 Building The Customer Signature Building a customer signature is a significant effort, but well worth the effort A signature summarizes customer or visitor behavior across hundreds of attributes, many which are specific to the site Once a signature is built, it can be used to answer many questions. The mining algorithms will pick the most important attributes for each question

Example attributes computed: Total Visits and Sales Revenue by Product Family Revenue by Month Customer State and Country Recency, Frequency, Monetary Latitude/Longitude from the Customers Postal Code Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 38 Migration Study - MEC Customers who migrated from low spenders in one 6 month period to high spenders in the following 6 month period Oct 2001 Mar 2002 Spent over $200 Spent $1 to $200 Apr 2002 Sep 2002 Spent over

$200 rs o t a gr i M Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA (5.5%) Spent under $200 (94.5%) 39 Key Characteristics of Migrators at MEC During October 2001 March 2002 (Initial 6 months) Purchased at least $70 of merchandise Purchased at least twice Largest single order was at least $40 Recommendation: Score light spending

customers based on their likelihood of migrating and market to high scorers. Used free shipping, not express shipping Live over 60 aerial kilometers from an MEC retail store Bought from these product families, such as socks, t-shirts, and accessories Customers who purchased shoulder bags and child carriers were LESS LIKELY to migrate Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 40 Customer Locations Relative to Retail Stores Heavy purchasing areas away from retail stores can suggest new retail store locations No stores in several hot areas: MEC is building a store in Montreal right now. Map of Canada with store locations. Black dots show store locations.

Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 41 Distance From Nearest Store (MEC) People farther away from retail stores spend more on average Account for most of the revenues Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 42 Other Results at MEC (See Appendix) Free shipping changed to flat-fee (C$6 flat charge) Orders - down 9.5% Total sales - up 6.5%

Gear Swap (buy/sell used gear) Visit-to-Purchase very low: 0.34% vs. 2.1% for non gear-swap However, these visitors converted to purchasing customers (over multiple visits) at a rate 62% higher than visitors who never visited gear swap! Visits where an FYI page (For-Your-Information) page was viewed had a Visit-to-Purchase conversion of 7.1% Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 43 Other Results at Debenhams (See Appendix) People who got the timeout page for a high percentage of their sessions are less likely to migrate (to heavy spenders) Revenue due to wedding list item purchases is clearly affected by summer weather Weddings are more common in the summer in the UK In June/July, 65% of revenues were generated through the wedding list

Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 44 Summary (I) E-commerce matches the needs of data mining Huge datasets (both rows and columns) Clean data (collected electronically) Very actionable (easy to do controlled experiments) Easy to measure return-on-investment Having a unified architecture (collection, transformation, analysis) saves much of the transformations needed (the 80% factor) and provides access to more data Customers need to crawl before they walk before they run. Must have simple reports Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 45 Summary (2)

Focused on specific vertical e-commerce retail Enabled us to write out-of-the box reports Easy for clients to get initial metrics and insights Encapsulate our expertise in this domain Focuses sales force, easier to demo with right vocabulary Provide visualization to show patterns (not discussed, but useful: interactive visualization) Many lessons, both at the business level and at the more data mining technical level to be reviewed by Rajesh Parekh Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 46

Resources WEBKDD workshops http://www.kohavi.com Mining E-commerce Data, the Good, the Bad, and the Ugly, invited talk at KDD 2001 industrial track Mining Customer Data, Etail CRM Summit, 2002 Integrating E-Commerce and Data Mining: Architecture and Challenges, ICDM 2001 E-metrics Study providing stats for multiple sites, Dec 2001

Applications of Data Mining to Electronic Commerce, special issue of Data Mining and Knowledge Discovery journal Real World Performance of Association Rule Algorithms, KDD2001 http://www.bluemartini.com/bi - case studies, live demo Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 47 Appendix Here are additional slides with some interesting insights Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 48 RFM Analysis RFM

Recency, Frequency, Monetary Example Insights from Debenhams Anonymous purchasers have lower average order amount Customers who have opted out [of e-mail] tend to have higher average order amount People in the age range 30-40 and 40-50 spend more on average Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 49 RFM Analysis (Debenhams) Recency, Frequency, and Monetary calculations are used extensively in retail for customer segmentation Implemented the Arthur-Hughes RFM Cube R, F, and M scores are binned into 5 equal sized

bins Each dimension is labeled 1 (best) 5 (worst) Interactive Look visualization using Filter Charts at charts instead of cell-tables Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 50 Complete RFM Majority of customers have purchased once Low Medium High Low Medium

High Recommendation More frequent customers have higher average order amount Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Targeted marketing campaigns to convert people to repeat purchasers, assuming they did not opt-out of e-mails 51 Interacting with the RFM visualization Explore sub-segments with filter charts Average Order Amount mapped to color People in the age range 30-40 and 40-50 spend more on average Low

Medium High Anonymous purchasers have lower average order amount Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 52 RFM for Debenhams Card Owners Recommendation Debenhams card owners Large group (> 1000) High average order amount Purchased once (F = 5) Not purchased recently (R=5) Low Medium High Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA

Send targeted email campaign since these are Debenhams customers. Try to awaken them! Low Medium High 53 Customers who have Opted Out Customers who have opted out tend to have higher average order amount Low Medium High Recommendation Recommendation

Send targeted emails to prevent email fatigue Log changes to opt out settings and track unsubscribes to identify email fatigue Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 54 Free Shipping Offer (MEC) Free shipping stopped on Aug 14, 2002 A flat $6 Canadian Dollars shipping charge introduced Express shipping at higher charge continues Observations Total sales -

up 6.5% Revenue (excluding shipping and tax) Orders - up 2.8% down 9.5% Average Sales per Order Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA up 18% 55 Free Shipping Offer (Cont.) The distribution shows fewer orders from low spenders (probably a good thing) No impact on rest of buyers

Fewer low spenders (<= $50) Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 56 Free Shipping Offer (Cont.) Breakdown of orders by shipping method More people used express shipping, probably because the delta to ship express wasnt as large (from C$6 instead of from C$0) Free/Standard Shipping Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Express Shipping 57 Gear Swap Pages (Cont.) Recommendation: Link back to MEC Shopping from Gear Swap Shop MEC Cycling

Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 58 Gear Swap Pages (Cont.) Done Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 59 Definitions for Gear Swap Analysis A visitor is defined as someone who is registered (MEC member) or is identified by a cookie Note that in the Blue Martini system a registered user will have all of his/her cookies combined into a single visitor ID Comparing visitors who viewed gear swap with those who did not Several non-bot sessions have 1 request that just visited the MEC homepage (Main/home.jsp) To get to the Gear Swap section you have to click at least twice

To make a fair comparison we have excluded all 1 request sessions that just visited the MEC homepage (Main/home.jsp) from the following analysis Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 60 Distribution of Gear Swap Visitors Visitors who viewed Gear Swap pages had a 62% higher visitor to purchaser conversion ratio as compared to those who did not view Gear Swap Visitors: Overall Visitors who ever viewed Gear Swap Visitors: X MEC members: Y Purchasing Customers: 14.3% of X

Z Visitors who never viewed Gear Swap Visitors: 85.7% of X MEC members: 20.8% of Y MEC members: 79.2% of Y Purchasing Customers: Purchasing Customers: 21.1% of Z Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 78.9% of Z 61 Distribution of Orders (the real ROI) Orders:

X Average Basket Value: $Y Overall Visitors who never viewed Gear Swap Visitors who ever viewed Gear Swap Orders: X Average Basket Value: 1.05 * Y Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Orders: 3,875 (78.3%) Average Basket Value: 0.98*Y 62

Distribution of Visits Although, Gear Swap visitors have lower visit-to-purchase conversion than non Gear Swap visitors, they visit more often and their overall visitor-to-purchase conversion is higher Overall Visits: X Visitors who never viewed Gear Swap Visitors who ever viewed Gear Swap Visits: 24.8% of X Visit to Purchase Conversion: 1.94% Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Visits: 75.2% of X

Visit to Purchase Conversion: 2.3% 63 Effectiveness of FYI Pages People viewing FYIs are more likely to purchase Viewed FYI Did Not View FYI Visits: 6.2% of all Visits: 93.8% of all Purchases: 23% of all Purchases: 77% of all Visit-to-Purchase: 7.1% Visit-to-Purchase: 1.2% Avg. Sales per Visit: 6.1X

Avg. Sales per Visit: $X Recommendation: Controlled experiment to study the effect of FYIs Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 64 FYIs (Cont.) Setting up controlled experiments to study the causeeffect relationship of FYI Select a handful of products (say 6) for introducing FYIs Randomly show the new FYIs to 50% of the visitors viewing these products and dont show the FYIs to the other 50% of the visitors At the end of the trial period (say 2-3 weeks) measure the visit-to-purchase conversion of the two groups Determine if there is a significant difference in the visit-topurchase conversion of the two groups Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 65 Debenhams Migrators: Timeout Some attributes are more useful when combined with other attributes

For each visitor we computed the number of sessions which went to the page timeout.jsp This was binned as shown on the X axis of the chart The height shows the number of visitors in each bin and color shows the percentage of those visitors who migrated Just looking at this variable alone it is difficult to tell what the pattern is Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 66 Migrators: Timeout By combining the number of timeout sessions with the total number of sessions for each visitor a pattern emerges In this heatmap the X axis shows the total number of sessions, the Y axis shows the number of timeout sessions, and color shows the percentage of migrators at each pair of values The green along the diagonal shows

that people who got the timeout page for a high percentage of their sessions are less likely to migrate Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 67 Migrators: Timeout The number of sessions a visitor has is a good indicator of whether or not they will migrate However there are some inconsistencies that are apparent. For example, why does the percent of visitors who migrate drop at 19 sessions? We can construct new attributes based on the relationship we saw between the number of timeouts and the number of sessions Two more attributes can be created: Number of sessions that did not time out Percentage of sessions that did not time out Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 68 Migrators: Timeout

Number of sessions without timeout is a good predictor of migration Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Percentage of sessions without timeout is also a good indicator of migration * 68,000 visitors with no timeout sessions have been filtered out 69 Distribution of Wedding Purchases over Time Revenue due to wedding list item purchases clearly affected by summer weather, when weddings are more common in the UK Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 70 Acxiom BMS supports ADN Acxiom Data Network

Seamless integration: get username/password Note: Acxiom recently changed their interface, so you will need a patch Comprehensive collection of US consumer and telephone data available via the internet Multi-sourced database Demographic, socioeconomic, and lifestyle information. Information on most U.S. households Contributors files refreshed a minimum of 3-12 times per year. Data sources include: County Real Estate Property Records, U.S. Telephone Directories, Public Information, Motor Vehicle Registrations,

Census Directories, Credit Grantors, Public Records and Consumer Data, Drivers Licenses, Voter Registrations, Product Registration Questionnaires, Catalogers, Magazines, Specialty Retailers, Packaged Goods Manufacturers, Accounts Receivable Files, Warranty Cards Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 71 Example - Income Graph showing incomes for a company that targets high-end customers based on POS purchases Income of their customers in blue The US population in red Note highest bracket

(30% vs. 5% for US) Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Percent 72 Setting Session Timeout (Debenhams) Debenhams set session timeout to 10 minutes to reduce memory footprint. 9.5% of visitors with an item in the cart lost it when they came back within 3 hours Look for upcoming article by us on developer summarizing this RFE filed to automatically extend sessions with carts Recommended timeout duration is 60 mins 2.5% of sessions with an item in cart will experience timeout RFE filed to remove bot sessions (oneclick) immediately to reduce memory footprint

Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 73 World Wide Revenue Detail Although Debenhams online site only ships in the UK, we see some revenue from the rest of the world. UK 98.8% US 0.6% Australia 0.1% Low Medium High NOTE: About 50% of the non-UK orders are wedding list purchases Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 74 Acxiom Integration Web

behavior is one axis Demographic information is another Blue Martini provides an extremely tight integration with Acxiom: Sign an agreement to get a password DSSGen will pull information from Acxiom over the internet as a part of building the data warehouse or as an option for an existing warehouse ZERO effort. No tapes, no customizations needed Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 75 Consumer Demographics Using Acxiom, we compared online shoppers to a sample of the population People who have a Travel and Entertainment credit card are 48% more likely to be online shoppers

(27% for people with premium credit card) People whose home was built after 1990 are 45% more likely to be online shoppers Households with income over $100K are 31% more likely to be online shoppers People under the age of 45 are 17% more likely to be online shoppers Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 76 Demographics - Income A higher household income means you are more likely to be an online shopper Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA

77 Demographics Credit Cards The more credit cards, the more likely you are to be an online shopper Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 78

Recently Viewed Presentations

  • Multiplying Signed Numbers - Math Rules

    Multiplying Signed Numbers - Math Rules

    Dividing Signed Numbers The same rules apply when dividing signed numbers. When dividing two numbers that are positive, you will get an answer that is positive. When dividing two numbers that have different signs, you will get an answer that...
  • Thinking Mathematically 13e

    Thinking Mathematically 13e

    Compose negations of statements. Translate between words and symbols. Interpret statements with quantifiers and form their negations. Find truth values of statements involving quantifiers and number sets. Statements and Qualifiers A statement is defined as a declarative sentence that is...
  • Database Replication in WAN

    Database Replication in WAN

    Writeset Containing modified tuples and their primary keys Implemented by triggers or log sniffing Currently not provided by DB standard interfaces PostgreSQL is currently implementing such interface Middleware based Data Replication providing Snapshot Isolation Yi Lin Bettina Kemme Marta Patiño-Martínez...
  • The gut-brain connection

    The gut-brain connection

    The "mini-Brain" Enteroendocrine cells. Release hormones. Synapse like structures. Electrical connection to vagal neurons (glutamate) Serotonin in the gut
  • Chapter 5. An Overview of Organic Reactions

    Chapter 5. An Overview of Organic Reactions

    CH 5: An Overview of Organic Reactions Renee Y. Becker CHM 2210 Valencia Community College * * Why this chapter? To understand organic and/or biochemistry, it is necessary to know: -What occurs -Why and how chemical reactions take place We...
  • Labeling, Interaction, and Crime: Societal Reaction and the ...

    Labeling, Interaction, and Crime: Societal Reaction and the ...

    Seldom one deviant act will provoke a strong enough societal reaction to create secondary deviance, unless the individual projects anticipatory fears onto the situation. Frequently, there is a progressive reciprocal relationship between the deviation of the person and the social...
  • Winter Counts (a closer look) PowerPoint II  You

    Winter Counts (a closer look) PowerPoint II You

    Winter Counts (a closer look) - PowerPoint Winter Count: Cloud . Shield (1788 - 1789) The winter was so cold that crows froze to death.? Click. Several counts mark this year when the winter was so cold, crows froze, including...
  • PERSEUS - West Creek Latin

    PERSEUS - West Creek Latin

    greatest center of the worship of Hera. Heraeum was the hill where the sanctuary of Hera stood. established by Phoroneus who decided in favor of Hera in the contest between Poseidon and Hera for the land. Poseidon dried up all...