By KK Davey

While innovation is crucial for any company that wants long-term growth, it’s become increasingly difficult to anticipate and meet shopper needs through new product introductions. A massive 80 to 90 percent of new product introductions fail to meet their goals. This means that marketers must constantly update their understanding of what customers want and need in order to be part of the sliver of new products that do well.

Steven Williams, PepsiCo’s Senior Vice President and General Manager, Global Walmart Team, invited me to moderate a panel of senior thought leaders from PepsiCo, Walmart and McKinsey on this very subject during the recent 2014 Enactus Partner Summit. The panel discussed some recent innovations in retail and consumer shopping habits as well as Big Data analytics. Here are some of my top takeaways for achieving product success, whether you are a CPG manufacturer or a retailer:

Consider micro-marketing – Chris Turner, Partner at McKinsey & Company, noted that successful marketers are increasingly planning at the very micro level, such as focusing on their top 100 cities instead of the mass market. Micro-marketing can take this even further with campaigns that have a highly personalized view of individuals based on their previous direct interactions with a brand, such as through social media, email and other digital engagement. Focusing on this type of granular growth can help companies make the right choices about where to compete.

Big Data offers bigger challenges…and bigger opportunities – Walmart’s Senior Vice President of Global Customer Insights Matt Kistler discussed his company’s Big Data challenges – specifically the depth of data they have been able to acquire. Walmart has access to almost 30 petabytes of customer data (one petabyte has 15 zeros!) so just imagine the extensive data mining required to find relevant data and then map it to meaningful consumer insights. But Walmart is doing it – they’ve moved beyond just capturing traditional point-of-sale data into social media, analytical applications and weather patterns, and they use that data to optimize local store assortment, tailor promotions and more. With Walmart’s latest efforts to apply Big Data to marketing as well as its mobile strategies, the company is continuing its trajectory of using Big Data to drive retail innovation.

Effective data analytics can uncover gaps in the market – Simon Lowden, Senior Vice President and Chief Marketing Officer for PepsiCo Beverages, outlined how his team created a plan that leveraged behavioral science and demand spaces (aka market gaps) to better understand consumers’ decision making process and set growth strategies for the future. This included a landmark study that explored the true drivers of consumption for the company’s brands so that PepsiCo could map shoppers’ path to purchase. One key insight was to encourage consumers to pair PepsiCo products when eating and drinking during the day, hence the company’s recent Better Together marketing approach.

Innovation starts at the store level – understanding the real drivers of purchase behavior – so marketers must know how to meet the needs of retail’s specific customers. (Millennials, for example, like to be communicated with differently, and they want more transparency and choices.) Also, with the vast amounts of customer data now available, there are huge opportunities for companies to focus on personalization and one-to-one marketing, even at the store level.

Marketers still spend a lot of time and effort on traditional insights instead of looking at and leveraging data analytics for key decisions and foresight. Finding and understanding these hidden insights can set up brands for a successful future, more than just focusing on what’s happening today.