Category Archives

8 Articles

Posted by Sonne Taylor on

The Spectrum Of User Surveys

Most advice about user surveys is really bad.

I know, because I’ve been part of the problem on this one.

Don’t believe me? Here’s a video I found and reuploaded from a few years ago. It’s in a folder with a very strongly worded readme all about what was going on at the time.

The trouble with advice like this, is that user surveys happen in one of two ways.

You’ve either got a highly mechanized feedback system (think coupon codes on the bottom of Fast Food receipts) or you’ve got an incredibly laid-back “user-interview.”

I don’t think it will be very controversial to assert that contemporary thinking has largely evolved to consider this a waste of time. (That is ‘coupin interviews,’) The process is too impersonal. The feedback is too inconsistent. If you’re still relying on this method, it might be well past time to start considering an alternative.

Still, the fundamentals are sound. The fine folks at ChartMogul put together a handy runthrough of NPS, I find myself passing along rather frequently.

There’s been a lot of great work generated on the topic of user interviews however. Eleonora Zucconi put together a fantastic collection of “46 Interview Questions For User Experience Researchers…” Teo Yu Sheng breaks out how you should think about asking questions in “5 Steps To Create Good User Interview Questions…” and I’d be remiss if I didn’t mention the vibrant discussion on Charles Liu’s “Never Ask What They Want

Learning to write in this way is a good skill to pick up, but it won’t keep you from making one of the biggest mistakes I see teams run into during the interview process.

Formulating your questions to get real feedback takes practice, but once you’ve figured it out it can be tempting to use a slightly different framing with each person you interview.

The trouble is, inconsistent surveying tends to generate inconsistent data.

That’s why it’s so important that you set your goals with the understanding that feedback happens on a spectrum.

In some circumstances, automated feedback may actually have some utility. For example, it’s probably ideal for “transactional” uses. In other cases (like say exploring a new feature roadmap) you might want to conduct something far less formal than a regimented user interview.

Trying to force each application into one predetermined feedback rail is a mistake. You won’t have the same answer every time, and that’s the point. Instead, try to focus on picking the right feedback mechanism for the task at hand.

The Spectrum Of User Surveys was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

Why You Should Pay Attention To Disney’s Take On This Generation’s iMessage Sticker

Early reactions to iMessage stickers weren’t great, but folks like Yono were quick to check it out. I’ll admit I’ve found myself unable to resist the temptation, and have also rolled my sleeves up to get into the fray.

It’s one of those things you get or you don’t — but when it comes to actually assembling the assets for an iMessage sticker pack? You could have gotten incredibly simply.

There’s a fairly well defined style guide, and the truth is most teams will have many of the necessary assets hanging out in a folder on the shared drive. It’s quite possible you could be looking at as little as an afternoon of work to launch your first sticker pack.

…or would have been.

In recent months a proliferation of image-only stickers has meant that the market has matured a little. If you’re paying attention to the top sticker packs you’ll notice that many include some degree of animation. Others rely on familiar characters.

But familiar characters aren’t enough. One of the earliest players to the space was the Disney company. Disney has a rich library of assets to work with and one has to look no farther than LINE to be given a rich opportunity to study best practices and learn lessons from earlier attempts with stickers.

Disney’s first collections of apps were simple. They included favorite characters with blurbs and static poses. They were generally priced at 1.99 (as are the majority of the apps on the platform.)

Reviews suffered.

Now this isn’t generally true, but I noticed this a few months ago with some curiosity and didn’t give it much thought.

That is until I took the opportunity to pick up one of the recently updated sticker packs I had had on my App Store wishlist.

That’s right: I bought Lion King stickers.

And while you can’t quite tell here, each one has its own animation.

It really is a great set.

So great, I turned around and grabbed one more.

Same thing.

Great pictures, great animations.

As I scrolled, I contemplated looking for a third pack, and wondered what other assets the team had converted into iMessage sticker form.

And then I saw that like all great Disney products, this one ended with a trip to the gift shop.

I can’t tell you from a distance if stickers are the right fit for your project. They might make sense if you have a particularly rich visual library or if you have something to say that can best be expressed with small shareable graphics.

It might sound as though they might not do much of anything for your business, and the truth is it will be a challenge to measure the impact they have. That being said, directed nostalgia like that can be a powerful thing to share with your customers.

While you can’t always influence the way your customers engage with your message emerging communications technologies like iMessage stickers can help to close the gap.

I can tell you anecdotally that I’ve spent more time with these stickers than I have with any of these underlying assets in years.

While that may not always translate directly into added sales, it has a real and perceptible value. Learning to measure and optimize for it is the trick to finding the best way to exploit a new opportunity like Disney has with these stickers.

Why You Should Pay Attention To Disney’s Take On This Generation’s iMessage Sticker was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

Sometimes, it’s nice to let the mouse win.

Sometimes, it’s nice to let the mouse win.

For the record though, they weren’t kidding.

Some Have Said, The Greatest Sign Off Message Of Known Internet History

You could actually probably get away with arguing that Team John Hanke started their anti-botting in full force Monday, but I’ve got to be honest with you, as someone who makes decisions about the value of media properties all day — I think you’ve done mathieu de fayet an incredible service with this project.

Ignoring that I’ve shopped my username into the screenshot (it’s a very natural way of watermarking a photo) by my approximation of the velocity of the queue system in place during the events described in the article, it would appear that a user requeueing throughout the day could run an average of 4.5 sessions a day (depending on the size of the queue in the evening, and how many sessions a player snuck in during overnight hours.)

Correct me if I’m mistaken, Dima Ryazanov, but on Saturday you guys had 121 Facebook Shares: 431 by the time the team pulled it?

When one considers these numbers:

Were all botters running the average of 4.5 sessions/day, this would look like one day of 100,000 DAU. Of course this wasn’t the case — and as is true in any user set, you’d expect to see this number compound over time.

Looking at Saturday where I remember 121 shares, let’s just assume all 100% of those users became 4.5 session a day people. You might expect then to be able to guess a potential number of “fake” users thusly — ->121*4.5=544.5

Looking at this, I could get lost for days. I haven’t gotten this close to a mobile app’s data in a long time. But that’s not what this article is about.

See, something else started popping up this week: rumors of Pokemon Go’s demise. Your data makes a pretty compelling case they too have likely been greatly exaggerated.

It’s a pretty open secret that some known percentage of digital user inventory is bogus. When advertisers like myself engage with a media company, we tend to try to price that in by reallocating media dollars accordingly. The truth is, bogus inventory doesn’t really move the needle for big players all that much. It hurts the small business folks a ton.

Nearly every way I’ve tried to look at it, your data seems to indicate that of the digital advertising platforms used most by small businesses and startups?

Pokemon Go likely well has one of the smallest fake user stats I’ve seen on a new platform.

That’s impressive work. props.

Sometimes, it’s nice to let the mouse win. was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

What Your Business Can Learn From The Kollywood Cafe

I’m going to show you something most marketers would keep to themselves.

S&T Traffic Map Post Redesign

For the last few weeks, I’ve been looking at this map a lot.

Some parts are easy to explain:

Domestic Traffic

Others are surprising:


There’s one part in particular I’ve been trying to figure out and so I’ve been spending a lot of time consuming content from the different regions where people consume my content. I figured that if I found anything interesting, I’d be able to write about it.

Savvy readers may notice, I’m doing that now.

But before I tell you about the Kollywood Cafe, I’ve got to explain one bit of context first. If you just want to read about the Cafe, make sure you skip to the section “Here’s Someone Who Did Something About It.”

An Overly Simplistic Way To Think About How Content Moves Online

If you spend much time online, you’ve probably seen a piece of media come at you from “out of nowhere.”

I’m probably not spoiling any secrets by telling you that this isn’t how it actually works — and if you don’t want to know about it you should just skip this section entirely.

What’s actually happening here is that people share links back and forth. If you build up a reputation online for sharing good links (or content about the content of those links,) you’ll attract a following of people who are interested in seeing those things. As you do more of it, it gets harder and harder to come up with “good content,” but because you’ve built up a reputation for having it, there are plenty of people who come out of the woodwork to help you keep it afloat.

On August third, Lori White published this. A day earlier (August 2nd) a content aggregating account, Top Stories reposted an article I’m sure you’ve seen. You may know it as the FAQ page from the fine folks at Little Free Pantry or as this June 1st article.

(If You Want Off Medium, That’s Here.)

Lots of people saw it, a lot of different ways.

In addition to looking at the timeline of the spread of a piece of content, you can pay attention to posts that suddenly have a lot more traffic than other posts. By doing that, you can easily identify the spot that you should pay attention to.

That’s how I noticed this.

Here’s Someone Who Did Something About It.

In “Now, Coimbatore has Tamil Nadu’s first sidewalk fridge to feed poor,” Komal Gauthman shares the story of how Kollywood Cafe took an opportunity to find a unique kind of corporate sponsorship.

Working with an NGO partner, Kollywood Cafe found an opportunity to put its leftover food to use.

They posted about it, to share information in their community — but they got something out of it for themselves, too.

Take a look at the number of reactions from that center post, the post about the food ATM. Compare it to the number of views on this article, an earlier bit of PR the company did.

I have just two questions.

Which one is doing more good for the world?

Which one is driving more sales?

When you act quickly on new ideas — new solutions to problems real people have, people notice. It’s not a gimmick, it works because it gives people something they are excited to support. It’ll (likely) work for every cause a person can reasonably believe a business might support — and the only trick is to make sure you present it in the right way.

What Your Business Can Learn From The Kollywood Cafe was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

How I Would Use Data To Open A Restaurant.

I’m a big fan of data driven decision making. In my job as a marketer it’s extremely important to separate myself from my work. After all if I’m trying to advertise a product that is clearly not aimed at my demographic, going with the kind of messaging or work I would like is just submitting to personal cognitive bias. Instead, I have to put myself in the head of the consumer. You have to do this at every step of the process, and even at the phase where you’re deciding what kind of company you even want to start.

I love the City of Denver. After moving here 4 years ago I still excitedly call it my home. One of the things that I love most about Denver is the diverse and growing culinary scene. From Rosenberg’s Bagels to Lo Stella Ristorante to Zengo, there’s so many great restaurants that have broadened my palate in many ways.

Want to know a secret though? If I was going to open a successful restaurant in Denver tomorrow, I wouldn’t base it on the foods I like but rather data.

Restaurants after all are a business. Personal preference doesn’t always translate to professional success. It may seem boring, but by using some simple quantitative and qualitative data we can make an informed decision about what is actually an underserved market in the Denver metro area.

So let’s first see where people are moving from. According to Zillow, 65% of newcomers to Denver are in fact from Colorado. Of the 35% of people not from Colorado the largest county that people immigrate from is Cook County, Illinois. (City of Chicago).

Now, qualitatively speaking there’s a few foods Chicago is known for. 1. Deep Dish Pizza. 2. Italian Beef. 3. Chicago Style Hot Dogs. The question becomes how does one decide which of these are the most popular? Well using social data we can infer some things.

We’re going to start off by using the world’s greatest free market research tool. The Facebook ad builder. Yes I am 100% serious. If you go into Facebook and begin to build an ad, before you pay for anything, you’re taken to an audience builder that will estimate the size of your audience in a geographic location before you buy on literally millions of data points. Now not everything will be on there, but this can help quickly see if there is a market available. For example, here’s the audience I’m going to start with for Deep Dish Pizza:

Here’s Hotdogs:

Finally, here’s Italian Beef:

As we can see, Deep Dish Pizza is quite popular, but Italian Beef is just on it’s heels. (I’m using Estimated Daily Reach). Let’s check out the competition in the area. According to yelp there are:

1. 141 places with “hotdog” on the menu

2. 71 places with “Italian beef”

3. 187 place with “deep dish”

Finally, we’re looking for qualitative data. I’m going to search on Twitter for the following keywords: Chicago, Denver, (name of food). Since I want relevant results, I’m going to accept items that appeared this year.

1. “hotdog”

There are no results during 2016.

2. “deep dish”

4 results of people asking for deep dish. Here’s an example:

3. “Italian Beef”

1 (very enthusiastic) result.

The results:

With audiences that are within 1500 people of one another, but with half the competition, I would personally choose to open an Italian Beef Restaurant. As the pace of migration from the Chicagoland area only increases, and with a lack of a Portillo’s or other iconic Chicago brand in the market, there is ripe opportunity to build and grow a restaurant that not only uses Chicagoans as a base audience, but also expands to Coloradans from other walks of life.

In the future, I’ll show you how to use just a few hundred dollars to test and validate whether these assumptions are true, again, using social media.

Tell me your thoughts in the comments!

How I Would Use Data To Open A Restaurant. was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

Gatherings In Open Air Retail Settings

If you’ve been paying attention to the massive number of articles on the topic this week, you’ve likely seen Randy Nelson’s look at the average time spent in the app. Close associates will know that we’ve [lovingly] questioned the validity of Hugh Kimura (SensorTower) public data in the past, but that’s generally been under the context of clients and teams trying to make App Store Optimization decisions without thinking through the problem. We love this article, and we wanted to share our own bit of foot traffic analysis that we think drives this point home.

ScreenGrab From Randy Nelson’s Article

A few days ago, we stopped by Osaka Ramen just off 3rd Avenue in Denver’s Cherry Creek neighborhood to continue our experiments with Pokémon Go. We found what we believed to be a triumphant explanation of what happens in that 33 minutes, 25 seconds.

If you’re not familiar with the in-game items in Pokémon Go, you’re probably baffled by the 33 minute number. If you are, you’ll know that popular items like Incense (an item that increases the spawn rate of random Pokémon) Lures (an item that increases the spawn rate of rare Pokémon for all players in an area) and Lucky Eggs all have a half-hour timer attached to them.

That suggests that an average play episode involves launching the app, using an item or walking around to a site where an item is being used, playing for the duration of that item and going on to something else.

We stopped by Osaka for lunch, and at 3:14 PM, we dropped a lure at the Bike Windmill Pokéstop off Denver’s 3rd Avenue.

Approximate Location of PokéStop

At the time we dropped the lure, there were already three or four players in the area hanging around the intersections. Once we dropped a lure, these players made their way to the spot, engaged with the game for a few minutes and then moved on to another intersection.

As near as we were able to determine, this pattern of stopping by spots while canvassing an area seems to be a popular play style.

In the first five minutes, three men and three women joined the group already touring the area and walked past our Pokéstop before continuing on through the region.

Over the next 10 minutes, five more women and one man stopped for a few moments by our Pokéstop before continuing on their way into one of the shopping centers of restaurants in the area.

As we approached 3:30, we noticed that two families (mothers and early teenage children) also stopped through the general vicinity of the Pokéstop. No stop lasted longer than two or three minutes — about the time it takes to catch a wild Pokémon in the game.

At 3:35, we noticed a woman we had seen before. She was in exercise clothes and circling the block. Each time she passed the PokéStop, she’d pause for two or three minutes, and then carry on her way, phone in hand.

Excluding the players that were already clearly in the area, our Osaka lure attracted 15 people in the time it was active between 3:14 and 3:44.

What we learned was that when you are including a PokéStop in a high traffic area, you’ll likely want to include some element of a promotion that increases staying power. If you don’t give players a reason to stop, they likely won’t — and will instead carry on in their play session.

Gatherings In Open Air Retail Settings was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

On Pokémon Go For Fun And Profit

For the non-marketer, our obsession over ROI is baffling. The truth is, it’s rather confusing to us, as well. Most ROI calculations aren’t particularly complex. Measure the number of people who could take an action. Measure the number of people who did take that action. Measure the impact of their behavior. Wash, Rinse, Repeat.

Of course this assumes that your business has a working understanding of each step of your sales cycle. The only way you can know the value of an action like a page like, is to understand what percentage of people who like your page move on to the next step of the process. Not all teams have invested the resources necessary to collect the data needed to have access to these insights. If you’re working on a team in that situation, you’re likely no stranger to basing your marketing decisions on statistical inference.

If you’re not a marketer, that probably all sounded like applesauce, but we hope you’re still with us, because this article is for you.

See, the truth is that the best companies with the savviest people have been working tirelessly to figure out how to convince the people they work with that real people are tired of scammy advertising. (Members of our team included!)

We know that people think marketers ruin everything. At S&T, we believe that bad marketers ruin everything — and it’s hard to tell the difference between the two. Good advertising is supposed to be fun. It’s a company’s chance to show off personality, and forgetting that has become all too common in our industry.

When we first learned of businesses using Pokemon lures to draw foot traffic, that’s what we saw. A way to share real fun with the people who support the teams we work to support.

So we aren’t going to start releasing our data by talking about the numbers or the conversion rates or any of the other things that marketers talk about amongst themselves. (Don’t get us wrong, we’re working to release real data, and you’re welcome to even help us out here.)

But we feel like we have an obligation to our team mates playing the game. That’s why we’re going to start by talking about an activity no business would ever support — gatherings in public places.

We conducted two experiments in popular gathering places in Denver.

On Saturday, June 9th at 3:50 we made our way to Denver’s scenic Cheesman Park. We dropped out first lure at 3:55.

The Pin Shows You Our Selected Site

In the first five minutes, players who were already in the park, began to congregate towards the site we had dropped out pin at. Four players found the lure immediately and made their way to the site.

A migration of any number of people across a park, is likely to attract attention, and by 4:15, we had 12 people gathered around the Cheesman Park Gazebo.

At 4:17, two other people happened on our scene. At 4:24 five more showed up. (That’s a total of 19.)

An Early Gathering Of Players At Cheesman Park

We were surprised by this as our lure ran out at 4:25. We left for our next site before we could see how many players hung around, but we had a great idea.

What if a higher density of lures was more likely to gather a crowd?

We Didn’t Wait Long To Find Out

On his way back from the office last night, our Creative Director noticed that a few players had placed a lure at the footsteps of the Colorado Capitol. Our whole team has been fired up about this project for some time, and so recognizing the opportunity to test our Cheesman hypothesis, Karl set about mining the Capitol building. At 9:09 all 7 Pokestops on the grounds had a lure.

Because of the size of the turf, we weren’t able to collect data down to the five minute interval level, and instead were forced to continually canvas the site.

From 9:09 to 9:49, we counted 58 people on the grounds of the Capitol.

At 9:49 when our Lure spread ran out, we were amazed to discover that two other players had dropped lure chips of their own to replace the two that had expired in front of the Capitol building. By 9:53, three chips had been dropped to replace the chips that had expired at the back of the capitol building.

That’s what we advertisers call viral engagement. It’s usually a good sign. It means that people are actually interested in the thing we’re doing, and they’re sharing that enthusiasm with the people around them.

After that drop, we counted an additional 9 players that were drawn onto the scene. Lures were once again replaced by the community of players at 10:12, and at 10:31 we noticed 8 more players had joined the throng.

Trainers At The Capitol

When the lures finally wore off at 10:55, we witnessed some 75 players make their way off the capitol grounds.

So, what’s the take away?

If you’re a player looking for people to play with, look for sites with a higher density of lures. If you’re looking to start a scene, you can easily do so with 1–7$ worth of lures, depending on where you’re playing.

If you drop a single lure in a site where people are already playing the game, you’ll likely concentrate those players in that area — like we found at Cheesman Park.

If you’re looking to get a gathering of a ton of people together, you’ll need to drop your lures in an area that people can easily get to. By creating a destination event, you’ll give people a reason to venture out and see what’s going on. Parts of town that have a high density of Pokéstops are a great candidate for this!

We think it’s important to remember that the point of a game is to be fun, and the point of a brand engaging with a game isn’t to sell the product. Get out there, and catch them all.

Over the next few days, we’ll be sharing more advertiser friendly selected findings from our experiments. We hope you’ll join us!

On Pokémon Go For Fun And Profit was originally published in Marketing Experimentation.

Posted by Sonne Taylor on

We Need Your Help To Measure The Return On Pokémon.

Those of you who follow us closely, know that for the past week, we’ve working on a study of the return on investment associated with the use of Pokemon Lures.

We’ve logged more than 35 hours of playtime — and we’ve spent most of it tracking other players.

Over the next few days, we’ll share some of the best practices we’ve collected as well as some of the raw foot traffic data for the collection of PokeStops we had access to.

We’re going to need more data that we have to make stable ROI calculations for a market outside of Denver. That’s why we want to make sure as many people as possible have an opportunity to contribute to this growing data set.

I’ve set up a google form that I’ll be monitoring throughout the day as we collect more field results.

We expect to start releasing some preliminary findings by mid-week.

If you’d like to help, here’s a link to the form.

We Need Your Help To Measure The Return On Pokemon. was originally published in Marketing Experimentation.