DEVOPS
Hacking Hacker News: Lessons Learned from a Security Researcher Wearing A Growth Hat
January 2, 2022
At Arnica, we strive to build Minimum Lovable Products (MLP). In order to build MLPs, we need to more deeply understand our users. As an early-stage company, we have fewer opportunities to talk with customers or prospects. So, we ran deep experimentation on HackerNews in order to understand what content and topics resonated most deeply with our users and make the most of each customer interaction.
Online traction and product-market fit are the holy grail for SaaS businesses. The summary of startup advice out there is to create a great product that people like, and to get people to know about it.
For young companies like Arnica, it boils down to delivering a Minimum Lovable Product that is derived from this research. In order to know what our customers would consider lovable, we want to hear them out, and while we had multiple validation calls with security, DevOps and engineering leaders, it doesn’t scale. Nothing replaces direct interactions with potential customers, but in order to “validate our validation calls” we must detach from our own bias towards our concepts and ensure we are data driven. This is the why we decided to embark on this path with this research.
Traffic alone can’t make a bad product sell, but if you have a great conversion metric, a great post can save months of organic growth and give you that much needed bump of early adopters.
Thankfully, Hacker News published a public API to make our job easy… to some extent. We decided to query a relatively small portion of the ~30M records and ended up extracting total 2,252,719 nodes and 4,028,273 relationships from the last 6 months into a Graph database, which is comprised of 110,801 users, 199,052 stories and 1,942,866 comments. Here is how the database is structured in this research:
The graph is pretty much self-explanatory except for one detail… what is “links_to”? Well, it is quite popular to link from stories or comments to others, so we decided to parse all titles and texts and link them to the relevant stories and comments. We found 13,126 links that helped us in this research.
Data segmentation and qualitative measures allow better classification of user and post popularity. Below is the approach we took in this research:
User Karma Classification
Story Popularity Classification
Story Comments Classification
Link Referrals Classification
Let’s look at the chart below. With the premise of Monday being the first day of the week, hence it is represented by the number 1 and so on.
Based on the data above, you get the biggest bang for your buck when posting a story on Tuesday and Thursday. These days are followed closely by Wednesday and Monday. Traction is low on weekends and relatively low on Fridays, which is typically a shorter day for many people.
With that said, the ratio between the top to high story scores, top to nice story scores, top to discredited story scores , and top to the total story scores represent the same popularity results. It means that posting on a specific day of week does not have an impact on the popularity of the story.
Let’s look at the chart below to understand how Top Story Popularity (defined above) is reflected based on the time the story was posted on.
From the chart above, we can clearly see that the top posts are published between 12–1 PM Eastern Time. The chart above is slightly different than the chart below with all story scores:
While there is some differentiation between the top posts and all posts, it seems like the general trend is to post between 10 AM and 1 PM Eastern Time. A couple of obvious conclusions we can derive from this comparison are:
Below is a table with a breakdown of the observed users’ karma vs. story popularity.
The highlighted values represent the top scored stories and the relevant user karma for the authors. The conclusion on this front is that users with karma above 30 have an equal chance to post a top scored story.
Before the data is explained, it is important to mention that links to a story are added after the story is posted. It means that the question in this case is how links to a given story impact on the popularity of it? Let’s observe the data below:
Based on this data, adding links to a story contribute to its score, but it cannot ensure a Top or High story score. As a matter of fact, Top ranked stories have a 5.25X higher ratio of links to story compared to High ranked stories.
The caveat is that linking to a story should not occur immediately. The average times between a Top story sharing and the link posting ranges between 5.5-13.5 hours (the 25th and 75th percentiles). By examining less popular stories and the links to them, the average time is ~10 hours. In other words, links to a story are typically impacting its popularity if posted within 5.5–13.5 hours after the story was posted.
Based on the data shared thus far, several tweaks can be made to increase the chances of a story popularity on Hacker News. With that said, good content is key to gaining more traction.
EDIT: thanks for the great feedback on Hacker News. Follow the discussion here.
