Some projects are rewarding for the journey. But when a project is full of grunt work, the reward can be the output.
In today’s output-and-insight-driven business environment, understanding your industry landscape is crucial for product managers. During his talk at WebExpo 2025, Daniel Do demonstrated how modern product managers can leverage web scraping and AI to gather relevant competitive data in a fraction of the time traditionally required.
The challenge of industry research
Daniel began the session by addressing a common dilemma faced by product managers: “How much time do you think that it takes to prepare an industry research report for your product?” His candid response, “It takes weeks,” resonated with many in the audience who had dealt with the similar struggles. This lengthy research process can hinder decision-making and slow down the pace of innovation within organisations. The challenge lies in efficiently collecting and analysing data to make informed strategic choices.
Daniel recalled his own experience when he ventured into the realm of product management software. He shared, “It took me weeks (last year) when I was analysing the product management software industry, when I wanted to co-found a startup,” highlighting the arduous task that many professionals undertake. However, he assured attendees that with the right frameworks and tools, this time-consuming process could be streamlined dramatically.
Unveiling the new tools
Introducing a smarter approach, Daniel revealed how web scraping and AI can dramatically speed up the research time. “With the framework that I’m going to show you today, you will cut the time from weeks to mere hours,” he asserted. The focus of his talk was on industry insights, specifically utilising platforms like G2 to gather data on product management software through its user reviews and company profiles.
Daniel pointed out that G2 serves as a “goldmine of information,” where product managers can easily compare companies within a specific market. He explained the G2 grid, stating, “On the Y axis, we have the market presence. Then on the X axis, there is the satisfaction representing the reviews and also the general rating,” which helps identify market leaders and focuses on competitors that warrant attention.
By employing a web scraping tool such as Apify, Daniel illustrated how accessible data collection has become. “For those who don’t know what Apify is, it’s basically a web scraping and automation platform,” providing attendees a glimpse into a resource that could empower them in their research efforts.
The power of AI in analysis
With data in hand, the next step becomes leveraging AI for analysis, which is a significant part of Daniel’s presentation. He shared that the transition from raw data to actionable insights is an exciting part of this optimised workflow. “What we will use is ChatGPT,” he emphasised, showcasing how generative AI can simplify data interpretation.
One of the most notable moments from this talk was when Daniel urged the audience to consider breaking down their data into themes. “We want to know what the themes are within that product management software industry just to know what is going on,” he told attendees, stressing the necessity of understanding market trends and user sentiments.
He effectively mapped out how well-organised themes, such as “ease of use” and “sprint and task management,” arose from the extensive reviews analysed. In one example, the data showed “The main problem is really that the tool is unintuitive, it’s very complex,” he explained, revealing significant user pain points that could inform product development strategies.
An actionable framework for product managers
As the talk progressed, Daniel shared how to identify an ideal customer profile (ICP) from the data gathered. “With the prompts that we used within this particular part, ChatGPT proposed to us that mid-market is way better than SMB,” he reflected, allowing attendees to understand the nuances of segmentation based on user feedback.
He zoomed in on competitors such as ClickUp, Asana, and Monday, walking through how to leverage the data to pinpoint where to focus marketing efforts. Additionally, he highlighted how switching reasons, as reported by users, can provide even more depth: “One that we identified is missing essential or advanced features,” he noted, painting a picture of the signals hidden within user feedback.
Towards the conclusion of his talk, Daniel insisted that this newfound capability to extract and analyse data does not merely enhance a product manager’s toolkit but transforms their role entirely. “You can basically right now with AI and web scraping do anything,” he affirmed, urging product managers to embrace these technologies as part of their skillset.
One of the most notable moments from this talk was when Daniel urged the audience to consider breaking down their data into themes. “We want to know what the themes are within that product management software industry just to know what is going on,” he told attendees, stressing the necessity of understanding market trends and user sentiments.
Cultivating insights for future success
Daniel’s call to action is quite simple: take advantage of the new capabilities offered by technology. He quipped, “There are unlimited possibilities of what you can do at that point,” encouraging the audience to envision innovative solutions they could develop by integrating web scraping and AI into their product management strategies.
For those seeking a deeper understanding, videos and slides from Daniel’s presentation are available below. The talk serves as a reminder that with the right tools, modern product management can be significantly enhanced, enabling agile, informed decision-making in today’s competitive landscape.