User Research in the Time of Coronavirus

Depending on where you live, the industry you work in, you may have experienced a collective escalation in the concern about Coronavirus, aka COVID-19. We certainly have here at User Interviews.
As research advocates, we’re always on team discover what’s true; don’t panic. What we know is that markets are reacting, and places of business are adjusting. Many conferences have been cancelled or postponed, and many teams are beginning to work remotely.We’ve been a fully remote team for over 2 years, and the slight majority of research our clients conduct is held remotely too. We’ve learned a few things about making remote work in general, and in research in particular, over the years, and now seemed like an especially useful time to round up all our best learnings on the subject.

The best stories about user research

Conduct Remote Research

If you’re concerned it will be harder to recruit participants for in-person studies in the current environment, remote sessions can be a great way to mitigate any lost research. Additionally, remote research sessions tend to have higher fill rates, lower no-show rates, and require lower incentives. We LOVE remote research if it’s a viable option for you, and even if it isn’t your typical go-to, there’s no time like the present to give it the old college try.

Running Amazing Remote UX Research Sessions with Sonya Badigian

This has been one of our sleeper hit episodes of Awkward Silences. Sonya Badigian has years of remote dominant research experience. Listen or read her top tips for making remote sessions as successful as possible.

3 Tips to Improve Your Remote User Interviews

In this article, Ritika Puri, also experienced in remote user interviews, offers three simple tips to make remote research effective.

The Very Best Tool for User Research Video Conferencing

A hallmark of remote user research is video conferencing software. Here we share our top pick for remote research. For more user research tools to round out your remote tech stack, check out our in-depth resource on user research tools, or our user research tools map.

The Ultimate Guide to User Research Incentives

When you cut commute time, the inconvenience of actually going somewhere, fighting traffic, or now, braving the potentially scary world of humans and germs, you’re fundamentally asking less of your participants, all things being equal. And asking less means you can pay less. You may be able to do more research in less time for less money. Turn those lemons into lemonade. Check out our incentives guide for specific guidance on how much to incentivize, depending on who you are recruiting, and how long your test is.

Working Remotely

If you or your team are new to working remotely, don’t feel like you need to start from scratch. As remote becomes more and more popular, there are better and better tools, frameworks, advice to make it work well for every kind of team.

The Hierarchy of Remote Work Needs

When you’re shipping product, not all user stories, requirements, what have you, are created equal. When it comes to working remotely, make sure your base needs are covered, and work your way up the pyramid of needs. If you’re new to remote work, don’t expect to find self actualization on your first day. Do expect to enjoy not commuting.

The Struggle (and Benefits) of Remote Work are Real

Remote work isn’t all ponies and rainbows all the time for everybody. Here we get real about the best and worst parts of remote work. Depending on how long you’re working remote, you may experience a large or small number of these, but we hope this will give you a sense of what you can expect.

The User Interviews Culture Handbook

Nearly a year ago we penned v1 of our culture handbook. Since we’re a fully remote company, remote work plays prominently in our culture. Here, among other things, we talk about how we make remote an asset to our culture, with tips you can steal on communication, and being effective.

Top Remote Work Resources

We don’t claim to be the very foremost experts on remote work yet. Here are a few excellent resources on remote work to peruse as well.

The Ultimate Guide to Remote Work by Zapier

The Remote Work Report by FYI

Remote by Basecamp

Last thoughts

While we hope life returns to “normal” as soon as possible, we’re also bullish on people, teams, and organizations getting a positive taste of the benefits of remote research and work, and maybe wanting to hold onto some of that more permanently. Put those hours of commuting into your work, family, or self. Turn those carbon emissions into…. fewer carbon emissions. Get more done. Live your best life.

Learn more about User Interviews.

Brian Cohen Receives the First Entrepreneurs Roundtable Community Award

Brian Cohen, champion of NYC tech, entrepreneur, angel investor, and author, was honored with the first ER Community Award on January 22, 2020. We’re fortunate that he’s been part of the Entrepreneurs Roundtable Community since we began hosting events in 2007.

As Founding Partner, New York Venture Partners and Chairman Emeritus, New York Angels, Brian’s impact on NYC tech is extensive. He’s a lifelong New Yorker who has championed startups and entrepreneurship, guiding young startups and helping them to understand the complexities of building venture businesses.

Our Host, DigitalOcean’s Hollie Haggans, Opens the Evening

brianawardhollie

Nearly 100 attendees mingled and shared stories of Brian’s insights, expertise, and endless energy.  Our host, Hollie Haggans, DigitalOcean’s Global Partnerships head, opened the evening. After a welcome from Murat and Jon, Murat MC’d the evening. Speakers honoring Brian included Owen Davis, Partner at Contour Ventures Partners, who delivered remarks about Brian’s many contributions to NYC tech; Comixology’s David Steinberger, who reflected on the immense value of receiving investment from Brian; and David Rose, Founder of New York Angels who provided a history of New York tech all the way from New Amsterdam to Brian Cohen. Murat and Brian closed the evening with a Q&A, followed by the presentation of the award.

Comixology’s David Steinberger

comixguy

NYC’s Entrepreneurs Roundtable Accelerator Announces Participants for its Winter 2020 Program; Companies Receive $100,000 Investments

New York—January 13, 2020—Entrepreneurs Roundtable Accelerator (ERA) today announced that it has selected its Winter 2020 class, which begins today. The companies selected to participate in the four-month program are innovating in a variety of significant industries, all with businesses ready to take advantage of ERA’s platform and the broader New York City opportunity.

This is the Accelerator’s 18th program. ERA’s initial investment in participant companies is $100,000.

ArtistOnGo is a marketplace that allows wellness and beauty artists to rent unutilized spaces in premium salons by the hour and work directly with their clients anytime, anywhere. Clients pay less, artists make more, and salons monetize unutilized capacity.

Coinapoly is an alternative asset management platform for fractional real estate ownership in global markets including India. Our technology simplifies granular real estate investing with our app to buy, sell, and manage properties. We deliver a better ROI by managing risk through transparency and liquidity.

FieldCLIX is a SaaS platform for wireless infrastructure installation projects. We help customers manage complex workflows for 5G buildouts and allow them to collaborate in real-time on project planning, field resource management, cost tracking, and more. FieldCLIX maximizes field productivity, improves project profitability, and helps accelerate infrastructure buildouts across the wireless industry.

Hailify is a mobility data company for rideshare drivers and fleets. Our proprietary data intelligence and infrastructure technology helps customers optimize utilization and identify high-demand pickup locations, and allows us to offer affordable rideshare insurance to customers. Hailify increases efficiency, improves logistics, and boosts revenue for customers.

Hazel is a personal care brand for women aged 50+. Starting with incontinence, we are reengineering disposable underwear – designing for fit, comfort, and style. We provide women a new option for buying and wearing underwear that is discreet, fashionable, and functional.

Mouth Off Health is a consumer wellness brand. Our first product is a dissolving gum that eliminates bad breath for up to four hours. We are innovating in how active ingredients are delivered with products that are convenient, environmentally friendly, and have superior efficacy.

Nayya is a software platform that enables insurance companies to better understand their customers. Using data science and machine learning, we allow insurers to more easily develop and customize new products while also identifying the right customer segments to sell them to.

Parento is an insurance agency that obtains and sells a proprietary insurance product for company-paid maternity and paternity leave policies. Our policies cover payroll expenses to let employers offer employees 100% of their salary from the day they begin their leave for up to 16 weeks. We help most employers offer paid leave and provide cost certainty, risk management, and parental leave administration.

RillaVoice is voice recognition software that captures insights from conversations shoppers have with salespeople in retail stores. Using our AI technology, RillaVoice analyzes conversations collected using lapel mics to better understand the store experience, demographics, conversion, and more. We give brick-and-mortar merchants a new way to analyze customer behavior that’s secure, compliant, and anonymous.

Salusion is a financial services platform for health savings accounts (HSA).  We advance HSA contributions to households with eligible health plans.  Salusion allows people who are financially unable to contribute to their HSA deduct their medical expenses saving a family of four an average of $700 per year.

Spotter is a software company for the long-haul trucking industry. Our AI model selects loads based on criteria including rate, schedule, and fuel costs to automatically find the best shipments for fleet owners and provide drivers with pickup and dropoff instructions. We help carriers reduce costs by automating repetitive tasks and increase revenue by improving load selection.

Top Corp is a SaaS company for brands to acquire opt-in data. Our technology lets brands collect privacy-compliant, first-party data via Topvote, our consumer engagement platform. Topvote uses branded incentives and rewards to run voting competitions in real-time across all channels, resulting in up to 5X increase in advertising engagement.

Undock is a SaaS platform for scheduling, coordinating, and hosting meetings. Our AI model automatically discovers the ideal meeting time for participants by comparing availability, preferences, and behavior. Undock makes meetings sharable and searchable, giving teams a single place for collaborative agendas, notes, and built-in conferencing.

About Entrepreneurs Roundtable Accelerator (ERA)

Entrepreneurs Roundtable Accelerator (ERA) is New York City’s leading technology accelerator and early-stage venture capital fund. It has invested in more than 200 start-ups since launching in 2011.  Its alumni companies, who come from all over the world, are already playing leading roles in the evolution of virtually every major global industry. To date, ERA alumni companies have raised more than $500 million in capital and collectively exceed $2.5 billion in market capitalization.

 

 

 

Forbes 30 Under 30 2020 includes alum from six ERA companies

We’re thrilled that the Forbes 30 Under 30 for 2020 included alum from six ERA companies. Here they are:

Founder conversations: Construction tech start-up finds inspiration — and a new strategy — in the trenches

Growing up in a Maryland construction family, Joe Leiva had mud on his workboots even before heading off to college for a civil engineering degree.

After returning to help with the family business, Joe saw firsthand how public works subcontractors were quickly buried by government red tape. Every contract required 20+ different forms, such as workforce demographics and daily reports, that were submitted throughout the course of a job. Worse yet, don’t follow the rules and you don’t get paid.

“I wanted to help these folks avoid compliance issues, to help shorten payment cycles because it’s so critical for these small businesses to meet payroll,” Joe said of his motivations on launching ProTrakr in 2015.

What got you here may not get you there

But the same industry knowledge gained from years of literally being in the trenches — “almost everything we did was underground” — also blinded Joe to what the market really needed, teaching him his first, hard lesson about building products: talk to customers frequently, even if you think you know what they’re going to say.

“We had a little bit of fire,” recalled Joe of ProTrakr’s early success, “so we just threw fuel on it without really taking a step back and questioning clients on what they really needed.”

But as growth leveled off for its project management suite — proving a complex sell to SMBs — a revelation appeared last year as several ProTrakr subcontractor customers were building Washington’s $5.6 billion Metro expansion.

A call came one day from an executive with the prime contractor coordinating the entire project. ProTrakr automated submission of work performance docs to the prime contractor for Joe’s sub customers, helping get their ducks in a row. But the prime had bigger worries.

“He told me, ‘How great it would be if all 200 of our subs were on your system, and I was just receiving data free-flowing in real time,’ ” Joe said. “That’s when the “aha!” moment came. I had always been looking at the problem from the eyes of the subs submitting to the primes, and I didn’t realize just how fragmented and disorganized it was for the larger companies.”

“We had to pump the brakes a bit and approach the bigger problem with a blank slate,” Joe said of a barnstorming effort to reset his team’s understanding of customer needs through dozens of interviews throughout the country.

The discovery journey quickly uncovered that subs and primes alike were relying solely on unsophisticated email and storage tools to keep track of the blizzard of required documents submitted on a regular basis.

ProTrakr already had the solution in hand, embedded in its larger platform. Launching a new, more targeted product meant carving out the document management module and giving it to all parties to serve as a centralized hub for all incoming file activities.     targetdocs_square

Product changes were just the beginning

Pivot is too strong a word for the change; Joe prefers “focus shift, as we are focused on the same industry, but we zeroed in on a critical need and tweaked our solution to meet it.”

And there were other benefits, such as a 6x expansion of target market beyond public works contractors to any company working on government-funded construction projects that require intensive document exchange.

As part of the shift, Joe renamed the company to TargetDocs and grappled with how to market its narrowed capabilities, since their solution didn’t cleanly fit into existing product categories — “there’s no construction lingo for it.”

And while the prior platform required a high-touch sales and implementation approach, the lighter-weight product is mostly self-service and relies a wider array of acquisition methods. There remains an upsell path to the legacy product for subs needing more power, but the sales team is squarely focused on the file hub. All of which required Joe to retool how TargetDocs did business, which proceeded relatively smoothly once the strategy was clarified.

“Hopefully we’ve landed with this change,” Joe said, “but this has made us much smarter going forward in so many other ways.”

Other pointers from Joe learned from his early days

  • Some of the best founders launch companies to solve problems they personally suffered through, but have the self-awareness and humility to know your experiences are simply one data point. “In my situation, being part of that industry gives you authenticity and gives you an emotional tie,” Joe said. “No way in hell would I have stumbled upon this coming from somewhere else. But even if you’re an expert in your field, try to take a step back to see the bigger picture.”
  • Understanding customer needs should be paired with gathering detailed data on spending patterns by customer segment. Identify the dollars you’re going after, whether it’s from incumbents or harvesting unallocated budget associated with solving new problems. “And when you look at the data, you’ll also see your marketing strategy and playbook start to emerge,” Joe added.
  • Always have an up-to-date model of your acquisition funnel, even if the data is sparse. As the TargetDocs looked at progression from lead to close on the original business, Joe said, “we started adding up the numbers, and it really didn’t make sense because it was too hard of a sell.” So when the new product signal came, TargetDocs quickly modeled how this would impact the funnel.

TargetDocs is a portfolio company of the Entrepreneurs Roundtable Accelerator (ERA), New York City’s largest and most influential accelerator that has guided nearly 200 US companies through their early stages with its impressive mentor and investor network. 

Scaling to multiple locations with Negotiatus

Heyday is a popular skincare company with locations in New York and California. After their doors first opened in 2015, Heyday experienced quick growth with a $3mm seed round followed shortly after by an $8mm Series A.

Maxwell Bennett, Operations Manager, is responsible for overseeing operations at the company, focusing on all aspects of purchasing and procurement in addition to other organizational and HR-related roles.


Choosing the right partner.

We wanted to cut back on spending… as well as having one go-to place.

By late 2017, it became apparent to Heyday that a scalable system was needed to manage increasingly complicated operations and finance processes. Heyday needed something that could better handle and visualize the multitude of invoices the company was receiving. “We used Google Sheets to track spending by location, and at that time we only were two locations,” Maxwell recalls, “We would submit the sheets to the CEO and the he would cross-check the credit card statements and the sheets.”

In evaluating systems, Heyday needed a partner that would assist with the invoice volume in addition to providing value on the purchasing and operations side. “We first looked at invoice management platforms,” Max describes, “However, the main deciding factor for Negotiatus came from us spending a lot on disposable goods like office supplies. We wanted to cut back on spending through the sourcing and substitution program and wanted one go-to place instead of having all of our purchasing managers manage a login for Amazon, WB Mason, Webstaurant and all the other places we order from.”


Seamless ordering for both supplies and retail.

First and foremost, it is a huge time saver.

Heyday immediately benefited from consolidated ordering across all their vendors ranging from their office and cleaning supplies to retail. “First and foremost, it is a huge time saver,” Maxwell says, “Going to one place for all the treatment supplies and office supplies – everyone knows exactly where to go. For retail alone, we have products from 10 different vendors, so instead of filling out order forms for each vendor for each location we can go to one place. Without Negotiatus it would have taken us three days to place all of our orders and now it takes just a couple hours.”

Controlling and adding to an approved product catalog was simple: “There are so many ways to link vendors and products… I love using the Chrome extension to add products on the fly. I can be on Amazon and just press the Negotiatus button to add a product right to my catalog or add it directly to my Negotiatus cart right there. And then it’s in Negotiatus and ready to order whenever we need it again.”


Visibility and organization.

It’s amazing for organization.

To better optimize processes internally, Maxwell and his team took advantage of Negotiatus’ real-time analytics: “Being able to see on the ‘Analytics’ page the breakdown by Cost Center, filter month to month or pay period to pay period, and look at our spend by location is so handy. Seeing the top products we spend money on is so helpful as well.”

Negotiatus also increased accountability within Heyday and made it easy to track down any purchases. “I am able to go into ‘Completed Orders’ and search for that one cabinet someone bought in February from Ikea, and I can easily look at the date, location, and who purchased it to quickly get to the bottom of any issue.”

“It’s amazing for organization,” Maxwell summarizes, “To see everyone’s orders in one place – to see and categorize our spend data – it is really great.”

Six ERA alum named to the 2019 Inc. 5000 “Most Successful Companies in America” list

We’re thrilled that Inc.’s annual guide to the 5,000 fastest-growing privately held companies in the U.S. included six ERA alum. Here they are:

Beware of Trees Falling on Your Home and Your Insurance Premium

Does Homeowners Insurance Cover Falling Trees? 

You’ve probably heard the old adage

If a tree falls in a forest, and no one hears it, does it make a sound?

But what if a tree falls on your house – are you covered? What about if it falls in your yard or falls on your fence? The answer is not straightforward.

Many Variables Come Into Play

This is one of those homeowners insurance answers where whether or not you’re covered does depend on the precise situation. And with 5% of homeowners filing at least one claim every year, you need to know what to do if you have to deal with a fallen tree. There are a lot of scenarios. So, let’s look at a few.

What if…

Your tree falls on your home or any other structure during a storm?
The national average for repairs after a storm is $7,296. The low end for damages may only be a few hundred while the high end can go to the tens of thousands. Wind damage is also the most frequent homeowners claim at roughly 25 percent of all claims.

Storms happen. And with any storm, debris removal comes with the cost of damages. If your tree falls on any roofed structure and causes damage to your property, then you’ll be covered by your homeowners insurance policy. Debris removal can be included on your policy, but sometimes carriers put a limit on debris removal (of $500, $1,000, or $2,000 per occurrence). Check your policy details for your exact coverage limit.

Your tree falls in your driveway?
Because you cannot use your driveway and because the tree may have damaged the driveway, homeowners insurance would cover its removal (in most states). However, check your deductible. Do you want to file a $1,200 claim with a $1,000 deductible – probably not.

A tree falls on your car?
Ouch. Your homeowners insurance would not cover it. Your auto insurance policy would under comprehensive or other-than-collision coverage.

Your tree fell down because of rot?
Homeowners insurance companies are a huge fan of responsible homeowners. Any time damage happens that could have been prevented, it gets classified as negligence. And when that happens, they most likely won’t cover damages. Something to keep in mind if you see a tree start to look like it could be a problem exercise some preventative maintenance to restore the tree or remove the tree before damage occurs. Supporting the tree with cables is risky, because if the cables break in a storm, the insurance company will say you were negligent in not removing the tree (given you put the cables up, you knew it had issues).

sailor moon fashion GIF

Your tree decides to fall into the street?
This is a tricky one. Either you or the city would be responsible for cleaning it up, depending on local laws in your county.

Your tree falls into your yard? 
While annoying, unfortunately, cleaning it up is on you. Homeowners insurance generally doesn’t cover trees falling over in your yard. They have to damage a structure for coverage to kick in. Some policies do, however, offer coverage for debris removal assistance.

Your tree fell because of…other causes?
If there was a riot, car accident, or fire that caused the tree to fall, then it would be classified under that peril and covered. On the other hand, if a tree fell over because of ground movement and damaged your house, it could be excluded if ground movement is an exclusion on your policy. (most common)

Your pesky neighbor’s tree fell into your yard?
This is where it gets unfortunate: where the tree lands determines responsibility. So, if the tree falls in your yard and it was healthy, you’re responsible.

If your neighbor was negligent, then their insurance could be responsible. (eg if their tree was rotting months ago, it should have been taken down, but they refused to do so)

Your pesky neighbor’s tree hit your house?
You’re covered. Phew. Many awkward summer barbeques in the backyard trying to avoid eye contact avoided. In some cases, your insurance company may have your neighbor’s homeowner’s insurance pay for any damages/removal through subrogation. And if that happens, you could recover your deductible.

How Much Coverage Will My Policy Provide?

Again, this varies from policy to policy. Generally speaking, the limit per tree is $500 for removal. Some policies limit the amount you can claim per event. For example, if your policy limits your claim total to $1,000 for tree removal, and it cost $500 to remove each tree, and 4 of them were knocked down, then you’d have to pay the extra $1,000 out of pocket.

Will My Insurer Buy New Trees?

Trees and other shrubs add value to your home. In some cases, it can add between 5% – 10% more value. That’s $15,000 – $30,000 on a $300,000 home. Wanting to replace them makes sense. Many policies have limits on trees and shrubs while some offer no coverage at all. Read your policy documents to see what coverage you have.

Keep in mind that if you sell plants as a business, then you would need to consult your business insurance for filing claims and getting replacements, not your personal homeowners policy.

Why Your Deductible Matters

Most common deductibles on a homeowners insurance policy are between $500 and $1,000. The average cost of tree removal can vary between $150 – $1,500 per tree. If it only costs $350 to remove the tree, then filing a claim would not be in your best interest.  Some home insurance policies have a $0 deductible specifically for the removal of fallen trees, so you may be in luck. Read your policy.

At your service,
Young Alfred

SEPARATING DATA SCIENTISTS AND SOFTWARE ENGINEERS 

Drawing credit: McCulloch and Pitts 1943.

Some organizations chose to leave the DS process alone and looked for paradigm analogies, like using demo as the DS-analogous operator to replace code review of SWE. They combined this with changing the way they think about problem-solving. However, the inherent issues of applying Agile SWE rigor to data science have been skirted.  We know that the major argument for leaving DS as a waterfall approach is that it’s inherently a holistic process which can’t, or shouldn’t be broken down. Several authors take the pragmatic approach of breaking up the process into research and engineering and applying Agile to the latter while leaving the former as a waterfall. Here we see the first references towards making the process scalable, i.e. allow more people to be added to a task to speed it up, which is a great goal. Specifically, these recommend a heterogeneous team consisting of data scientists, data engineers, and software engineers.  The argument, again pragmatic, is that it is nearly impossible to find engineers capable of both performing in both research and SWE roles. Pragmatism is always the right way to go for a startup, but a chain is only as strong its weakest link.  Having a process by which product engineers need to wait on a waterfall DS task to complete before we can start incorporating results into the product leaves us feeling uneasy at TWOSENSE.AI.

AGILE PROCESSES FROM DATA SCIENCE

There are some great pieces shared from the trenches of early attempts.  One of the first pieces of learning is to keep individual DS tasks focused on a scope by the type of work that’s being done, e.g. literature review, data exploration, algorithm development, result analysis, review, and deployment phases for each sprint or story [RECOMMENDED READ]. This is a great piece of insight because it shows us their approach to actually breaking down a big thing into smaller steps and tasks as it relates to DS. We also see the concept of skipping or reassessing the remaining tickets based on what we learn during execution.  This concept is so exciting because it addresses one of the largest concerns of Agile ML critics: you don’t know what you don’t know. The big takeaway here is that you need to be continuously adjusting throughout the sprint based on what you’re learning.  This goes against mainstream Agile SWE wisdom since it makes projects more difficult to plan.  However, it’s a pragmatic approach to planning yet incorporating the intrinsic component of not knowing what you don’t know. Trying to enforce strict adherence to the plan in the face of new understandings feels to me like it would subtract from focus on the sprint goals, like dogma instead of pragmatism.

With this dynamic process, it’s important to stay on focus and make each ticket or task something with a tangible benefit. A great piece [RECOMMENDED READ] articulates the iterative nature of this process, starting off with a baseline and iteratively improving it with each task or ticket.   With a dynamic process that includes many unknowns and iterative improvements, it is also dangerous to be overly focused on story points [RECOMMENDED READ]. Doing so here can encourage gamification of the process where engineers optimize for the human performance metrics at the cost of ML performance. It’s important to evaluate the benefit of a small iteration, and the context switching cost of picking this back up later at another iteration. The same could easily be said about Agile SWE as well, however, the difference here is that DS/ML tasks are about answering a question far more often. One of the ways to address this is to acknowledge that there are some jobs that just don’t have deliverable code results [RECOMMEND READ] that should still be reviewed but to have these be shared as show-and-tell or demos.

FROM AGILE DATA SCIENCE TO AGILE MACHINE LEARNING

So far, we’ve looked at Agile DS and gotten as much as possible from the literature that seems relevant to MLE as well.  One thing to keep in mind is that the vast majority of what we’ve been speaking about thus far was designed for very long timelines.  The article above was speaking about sprints on a timeline of about 1 year (!!), so while many of the lessons learned are still relevant, there may be quite a bit further yet to go to get things in line with a 2-week sprint.  Furthermore, they’re looking at DS, which is not the same as MLE. DS is about finding questions the data can answer and answers to those questions where the consumer is most often a human. A DS sprint is therefore focused on answering a question.   MLE is about starting with a problem independent of the data and building an ML solution to it, where the consumer is usually a piece of software or an actuator of some kind. 

AGILE FOR MACHINE LEARNING ENGINEERING

When we dive into what’s been published on Agile for MLE, the literature is much thinner. A company called YellowRoad (I’m not sure what they do exactly, but I’m sure they do it really well!) has put out some great content.  They propose starting off with quick solutions over optimal ones [RECOMMENDED READ] as a way to make things iterative yet shippable. Google famously “blur the line between research and engineering activities” and say just about the same thing. <CONJECTURE> There is a vast chasm between knowing what the best solution is and implementing that in a product.  Bridging that chasm is the hardest part, not just in DS/ML, but in startups, art, everything: it’s called “execution.” Trying to find a process that closes that gap and unifies research and product dev is really hard, but you only have to invest that effort once. It’s what we’re trying to accomplish with all of this here. Having to bridge that gap every quarter, every sprint, or even every ticket feels like an ongoing cost that will be fatal for a startup’s execution. I think that this is also what Google discovered, and why they’ve made the decisions that they have.</CONJECTURE>

Google also says [RECOMMENDED READ] (Rule #4) that it’s better to start with a bare-bones model (we call it a skeleton) and focus on building the infrastructure first, then building out the ML iteratively once that’s in place. They even go a step further (Rule #1) and say try and start off with an expert system as your baseline to focus on the infrastructure alone before you even get into ML at all.  Those rules are fantastic, I recommend at least scanning all 37. On top of that, they have a great paper on ML technical debt [RECOMMENDED READ] that while slightly off-topic here, will greatly help anyone on the same journey with us.  It also concludes by saying that even if your team isn’t made up of full-stack ML engineers, these processes and the technical debt affects both SWEs and DSs, and there needs to be buy-in from both sides and tight integration between them.

In a recent conversation with Sven Strohband of Khosla, Sven stated that every ML/AI company he’s worked with that was able to execute successfully followed a “Challenger Champion” approach to MLE.  This is a concept that stems from the early days of financial modeling (as far as I can tell), where an incumbent system (the Champion) is A/B tested against a proposed improvement.  I love this concept because when combined with rigorous evaluation metrics, it creates a form of end-to-end testing where the metrics are the test.  It also creates a Kaizen culture, which is one of my favorite concepts, where small changes that improve the whole are adopted over time.

CONCLUSION

Before designing our process, we read everything we could find on the topic.  Here’s what we took away.

The TL;DR is this. Agile is a way of thinking more than anything else as it applies to Data Science and Machine Learning. It’s about creating a process that breaks big things down into small tasks, continually readjusts those tasks for changing requirements, and ships as often as the product does. Make it scalable starting off with the bare minimum to touch all components, and iterating from there on the different pieces. Try to beat the performance of the incumbent system with your “challenger” improvement by measuring against performance metrics that are relevant to you. Watch out for overly focusing on story points along the way, and use demos to show non-shippable results.  Finally, if you want to work in an Agile way, you’re going to need a process that combines Research and Software Engineering efforts into Agile Machine Learning Engineering.

When Are You Ready for Multi-Touch Attribution?

Now you may be thinking: Does it make sense for me to steer away from GA and implement multi-touch attribution?

Not everyone is ready for multi-touch attribution — it’s not just a box to be checked. Think about it like buying a house: it can be a big step towards financial success, or it can be a leap into instability.

Here’s what being a good MTA candidate typically looks like.  

1) You’re marketing through multiple channels. Multi-touch attribution won’t be much of a value-add unless you’re marketing through multiple ad platforms. At this point, your customers’ path to purchase is less straightforward — you need a sophisticated way of informing budget allocation across ad platforms. If you’re not marketing through multiple platforms, single-touch attribution is probably effective enough.

2) You’re considering or actively buying offline. The moment you do offline marketing is the moment GA becomes inadequate for ingesting relevant data. If you can’t measure your offline marketing, how will you if you should increase or cut back spend here?

3) You’re spending more than $20K a month on advertising. If you’re not spending at least $20K across ad platforms, you have bigger priorities than implementing MTA. At this stage, your time is better spent figuring out which ad platforms work best and improving on these so that you can get your ad budget higher.

4) There are at least two marketers on your team. Ideally, there’s someone on your team dedicated to MTA. This usually isn’t feasible if you’re the only marketer. There’s a lot of work involved: You’ll need to carve out time to define KPIs, ensure all the relevant data is ingested and establish a QA process. Once attribution results come in, you’ll need to check that activities captured in MTA match the data from your ad platforms and that the results make intuitive sense.

5) Your team has realistic expectations. MTA benefits are rarely realized overnight. Everyone involved should understand that you’re building the necessary marketing foundation for the next couple of years, not tomorrow.  It’s likely that for the first few months, your campaigns will be running without any MTA optimizations in order to establish a baseline MTA benchmark.

You wouldn’t buy a house without planning for how it will get furnished, renovated and turned into a long-term asset. Similarly, you wouldn’t invest in MTA without planning for what data goes into it, how it will be maintained and how it will add long-term value.

When you’re actually ready for MTA, you’ll likely find it to be a necessary supplement to Google Analytics.

Learn more about Rockerbox.