This was a chat assistant for an e-commerce website that interacts with users in natural language and helps them find fashion items (apparel, accessories, footwear etc.) to shop for. The developed system can intelligently understand what the user is asking for, query the website's database, throw in some personalised recommendations, factor in current trends from social media, and respond to the user in natural language with a list of products.
Expected Outcome
This project was part of a hackathon I participated in. The hackathon was organised by an e-commerce company and we had to build a chat bot that would converse with the users in natural language, understand what items they are asking for, fetch relevant items from the database and display them to help the user make a purchase.
In addition to all that while deciding which items to recommend to the user, the system also does the following:
- Periodically capture current fashion trends from social media, and factor trendiness of items in recommendations
- Looks at user interactions (clicks, add to carts), and past purchase history for capturing personal tastes of the users
Team and My Role
I worked alongside another developer for this project. The project was divided into three major modules:
- Trendiness Extraction & Calculator
- Personalised Recommendation Engine
- Natural Language Chat Interface
I architected and them implemented the current trends extractor & calculator and the natural language chat interface modules. My teammate worked on the personalised recommendations module, and the React UI.
Tech Stack and Explanation
- React
- Node.js & Express
- Google Cloud
- LangChain
- OpenAI Platform
I picked Google Cloud and OpenAI Platform because of their high quality AI services. Their combination unlocks some truly unreal potential. Also Google Cloud gave us a very liberal $400 free trial.
LangChain is a library that assists in making complex and powerful apps that rely on large language models (LLMs). It provides useful abstractions and wrappers for functionality that LLM based apps commonly require.
Research
As this was a big project it demanded a significant amount of research before beginning to write code.
For instance for calculating trendiness of store items I knew I had to scrape fashion images from social media. I chose Instagram because my online research indicated that that is the most popular social media app used by people to make their fashion purchase choices.
For getting data from Instagram I tried the official API, and a third-party Instagram web reverse engineered API client library. I could not use the former because the approval process for the official API would take more time than what we had for the hackathon, and the latter just got my test account banned. I finally settled with an unofficial API called Lamadava which relies on an intricate network of proxies and bot account infrastructure to scrape public Instagram data. It was a cheap choice too which cost just $0.0006/request and you could get 30 (approx.) posts per request. That came out to 100,000 posts scraping with our $2 free trial.
Similarly for finding which items in the store database are similar to the trending items scraped from Instagram I decided to use Google Cloud's Product Search API.
Architecture
The trendiness calculator's and the recommendation engine's output were combined by using a simple mathematical function to calculate a "purchasability score". The chat interface then relied on that score when making product recommendations to the user.
Trendiness Calculator
The purpose of this module is to assign a trend_score to each item in the shop.
Obtaining Trending Fashion Data
I made a curated list of top fashion related hashtags, and top fashion influencers and decided to scrape images from them.
I scraped the current top 27 posts for each hashtag, and most recent 33 posts for each influencer. For video posts and reels, I took the thumbnail image instead of the video file.
Trend Score Calculation
I came up with a simple algorithm for calculating the trend score for each item in the shop.
- Initialise a dictionary of a
trend_scorefor each item in our store withtrend_score[i] = 0 ∀ i, wheretrend_score[i]is the trend score for the store item with idi - Detect individual objects in each of the scraped images, and crop detected fashion items (caps, tops, shoes, earrings, necklaces etc.) from the images
- For each detected fashion item do the following:
- Find top-n items similar to the detected item, in the store items dataset
- For each such similar item found do
trend_score[similar_item_id] += similarity_score
- Group by category (caps, tops, shoes, earrings, necklaces etc.), and then normalise the
trend_scorevalues of items - Repeat the above steps every
kdays/hours, depending on how frequent trends you want the system to reflect
Following Google Cloud services were used while implementing the above algorithm:
- Object localisation: Used for identifying individual fashion objects (caps, tops, shoes, earrings, necklaces etc.) in the images scraped from Instagram
- Vision product search: Used for search for similar products. This system was fed cropped fashion objects which were generated using the output of the object localisation service. This was utilised to get top $n$ similar items in store dataset for each item in the scraped images
LLM based Chat Interface
OpenAI Platform's gpt-3.5-turbo-0613 chat model served as the backend for the chat interface. LangChain provided composable components for common functionality and convenient wrappers around OpenAI's API.
Two LLM models interacted with each other to intelligently extract users' requirements from their natural language chat. You can see the details in the diagram below.
OpenAI's gpt-3.5-turbo-0613 function calling feature was used to make the LLM call functions.
My Learnings
I could write an entire essay on the learnings I gained from this project. Most significantly, I learned how to efficiently perform and deliver in a time-constrained and high-stress environment as we had only 12 days to complete the project. That including everything from the ideation/research to finishing up a working prototype.
I also learned how to use LangChain and make truly powerful apps that rely on LLM and generative AI functionality. I also learned my way around Google Cloud, specifically I worked hands-on with the compute engines, storage buckets, and the vision ai API.
As this was my first time working with both LangChain and Google Cloud and I had to learn them on the job and deliver a product.