awesome_raman

By the way, the technical chapter names are inspired by the name that Docker assigns to their containers. The name generator code can be found here if you are curious.

One afternoon in the summer of 2017, I was sitting at home in Sunnyvale and telling my friend Karthik that BuiltWith is every developer's dream.

A data product developed by just one person. It was one of those glitches in the Matrix.

Until then I had not heard too many stories about one-person startups or one-person companies. BuiltWith seemed like a product that sounded like a fairy-tale.

One person finds a problem that needs to be solved. And after putting in the necessary effort has a product that works and that other people are willing to use for a fee.

I started to read more about one-person companies and found many products that seemed to have a similar origin story as BuiltWith.

A lot of the products I came across fit a pattern. The pattern can loosely be described as the repackaging of publicly available data.

Free, publicly accessible web data collected and transformed into a form that increases its business value.

There are a large number of products out there just like BuiltWith.

NerdyData, PublicWWW, RivalScan, ShopRank, DataNyze, WebSpotter are a few I’ve come across. New ones keep appearing regularly.

The value of these products is to make it easy for their target user to get their job done.

The target users are sales people,lead generation specialists or competitive intelligence analysts. Users who benefit from such tools may also come from the finance and private equity businesses. And these are not the only professions who may wish to use such tools. As a web developer I would be very interested in knowing which Javascript technology is on the rise.

But what if you are a Data Scientist or Machine Learning Engineer. What are ways you can figure out which Python library is gaining in popularity?

We will discuss some of these usecases in later chapters when we build our own product with public data.

Coming back to BuiltWith.

Any developer could build something like BuiltWith, right? Take large amounts of publicly available data, transform it and deliver it to the target users and laugh all the way to the bank.

Karthik had a way of agreeing while disagreeing. While we did agree that it was possible, we certainly did not think it was easy. Karthik mentioned the business aspects of marketing, sales, customer service, competition, addressable market, TAM, and so on.

I acknowledged that the business aspects required more than a developer’s mind and skills but I was only focused on building a small enough product to enjoy the thrill of doing it end to end.

Sales and Marketing of any product is as specialized as developing technology solutions. As a technology professional we often believe that if we build it, they will come.

This is perhaps true in very rare cases but for the vast majority of cases, the process of building starts first with figuring out if a need exists in the market for what you wamt to build.

This blog post discusses the if you build it, they will come nicely.

And talking to the market or potential users isn’t the first muscle that a technologist wants (or knows how) to flex.

While I agreed with Karthik that in a crowded space that BuiltWith and others were operating in, another product would have to jostle for attention with marketing and sales being a very important aspect.

But, I also made it very clear that I all I was trying to do was build. And the only thrill I was seeking was to see my ideas come to life.

That got Karthik interested. I told him that I didn’t have any agenda or any specific outcomes to drive with this project. On the contrary, I wanted to work on it for indefinite amounts of time and preferably on weekends.

After nodding together for a few minutes, it was decided.

I was going to put on hold my current side project and take on this one. At any given time most technology professionals are tinkering with multiple side projects.

And if you don’t have one, the best thing to start with is learning a new programming language. God knows I’ve tried to learn a Lisp-like language for over a decade. I started with Clojure and gave up after fiddling with the IDE setup. And the reason I got inspired to learn a Lisp-like language in the first place was after reading Hackers and Painers by Paul Graham.

If you want a quick introduction to why PG thinks Lisp is awesome, you should read this article

After deciding to get started, I had this feeling that I had finally found a side project that would actually stick. Maybe it was because we were hanging out in my garage and living in the California where garages had given birth to top Internet companies.

In the next chapter I talk about the allure of garges and humble beginnings of some of the most storied companies in the world.

Building my own data product end to end was certainly inspiring but the garage sessions with Karthik were always very inspiring and kept me motivated to keep working on the project.

We did a laundry list of items that came to mind as we started a brainstorming session with a second cup of chai and rusks.

• BuiltWith does a lot of things. We will only focus on a specific feature, say identifying JavaScript libraries.
• Find a list of websites to crawl.
• Don’t try to crawl as many websites as BuiltWith. Just crawl a small subset.
• Store the data locally, say in SQLite.
• Don’t chew more by trying to use the cloud or unnecessarily adding more components to the mix.
• Premature Optimization. Don’t do it.
• Build UI using libraries in Python itself, like Plotly Dash or Streamlit.
• Don’t try to learn new languages or frameworks.
• Stick to Python.
• Use Docker and VSCode to keep things organized.
• For many of my side projects, I used to just keep everything in the one_wonderful.py file.
• For scaling use Cloud Run on Google Cloud Platform. We also just wanted to play with Cloud Run.
• Keep reminding yourself that this is a side project and not your ticket to early retirement.

I was already planning which island in Hawaii I was going to retire to.