Unveiling Customer Insights & Complaints from Social Media with Gen AI and Advanced Analytics

Palistha Deshar


Share

We scraped Twitter for insights on Indian Railways, utilizing Selenium for web scraping and a Large Language Model (LLM) for tweet classification. The project analyzed complaints from 2014 to 2024, revealing common issues like delays and cleanliness. This data helps Indian Railways improve services based on passenger feedback.

Github repo link: https://github.com/RamailoTech/mle-scrapers.git

The Indian Railways is one of the largest railway networks in the world, serving millions of passengers daily. With such a vast user base, social media platforms like Twitter have become a popular medium for passengers to voice their experiences, complaints, and suggestions. To harness this valuable data, we embarked on a project to scrape tweets tagging the official Indian Railways Twitter handle, classify the tweets to identify complaints and derive insights from the data.

The following steps were involved:
  1. Utilizing Selenium for Web Scraping

    • Selenium was chosen for its robust capabilities in automating browser interactions, making it suitable for scraping dynamic web content from Twitter.

  2. Navigating to the Search Results Page

    • Use specific search queries to find tweets mentioning Indian Railways. (Example: (@RailMinIndia) since:2024-01-01 -filter:links -filter:replies)

  3. Scraping the Tweets

    • Scrape the tweet text using Selenium.

    • Implement automated scrolling to load more tweets until the desired amount is reached.

  4. Data Cleaning and Validation

    • Filter out unique tweets.

    • Filter out tweets that are null or have a word count of less than four.

    • Clean the tweet text by removing unwanted spaces and normalizing the text.

  5. Designing a System Prompt for Classification

    • Develop a system prompt to classify tweets using a Large Language Model (LLM).

  6. Classifying Tweets Using LLM

    • Incorporate a large language model (e.g., GPT-3.5 turbo) using an API.

    • Use the LLM to determine if a tweet is a complaint.

    • Classify complaint tweets into specific categories such as delay, crowding, expensive, derail, hygiene, and management.

  7. Saving Data

    • Save the scraped and classified tweets in a structured format.

Results

The project successfully scraped and classified around 200 tweets for each year from the year 2014 to 2024 tagging Indian Railways. The analysis revealed common areas of concern such as train delays, cleanliness, and customer service issues. The insights gained can help Indian Railways address passenger grievances more effectively and improve their services.

Go through the GitHub repository to see what an output looks like. (link)

Challenges and Key Learnings

  • Handling Dynamic Content: Twitter’s dynamic loading of content posed a challenge. Selenium's ability to automate scrolling and handle dynamic elements was crucial.

  • Rate Limiting and Captchas: Frequent login requests triggered CAPTCHAs. Implementing delays helped mitigate these issues.

  • Multiple Logins Leading to Account Blocks: Using multiple logins resulted in account blocks. Strategies to manage and rotate accounts were necessary.

  • Deeply Nested Divs: The complex structure of Twitter’s HTML, with deeply nested divs, made it challenging to locate and extract the necessary elements for scraping tweets. Precise and dynamic XPath or CSS selectors were essential.