Last Updated: Wednesday, August 28, 2024 10:39 AM - Data up to Sunday, 28 January, 2024
Sentiment analysis was carried out using textblob, Llama 3.1b LLM and further supplemented with a hand tuned final review, this data was kept separate allowing a comparison to be drawn between the three methods. It will come as no surprise that understanding the intent behind a headline without any further supporting information is quite challenging for automated libraries such as textblob or general LLMs such as Llama 3.1b.
A future project will include training an LLM on this specific task, however there's not enough data at this point to make this worthwhile, I am currently running a bot scraping the headlines, which I plan on running for twelve months.
In the process of manually annotating the dataset, I must acknowledge the possibility of introducing personal biases that can influence the accuracy of results. Specifically, I may be susceptible to three types of cognitive biases: confirmation bias, anchoring bias, and availability heuristic.
Confirmation Bias arises when existing beliefs or opinions influence the perception of new information, leading to the subject selectively focussing on evidence that confirms our views while ignoring contradictory data. In the context of sentiment analysis, this could result in consistently labeling samples as positive or negative based on personal opinions, rather than objective criteria.
Anchoring Bias occurs when initial classification or impression influence subsequent judgments, even if the new information doesn't necessarily match the original criteria. For example, initially labelling a sample as "positive," means that similar samples may be labelled as positive as well, potentially introducing consistency errors into annotations.
Availability Heuristic Bias can also affect the annotation process, leading to overemphasis of extreme cases (e.g., strongly negative or positive sentiments) and overlooking more nuanced opinions. This bias can result in an inaccurate representation of the overall sentiment distribution in the dataset.
To minimize personal biases during classification, identifying information that could influence interpretation was hidden from the view. This includes publication title and date, which helps to ensure that the classifications are more objective and less influenced by individual perspectives.
Furthermore, the sort order of headlines was randomised to reduce the potential for Availability Heuristic Bias. A significant number of headlines began with the phrase "I'm a .." or "I / I've ...." indicative of a personal experience or subject matter style of headline preferred by certain media outlets (usually with a negative sentiment bias). Randomising the order of headlines, spread these problematic headlines out with the goal of reducing personal bias still further.
| title | Negative | Neutral | Positive |
|---|---|---|---|
| hand tuned | 57% | 9% | 34% |
| Llama | 51% | 28% | 21% |
| TextBlob | 28% | 34% | 38% |
| ChatGPT | 18% | 75% | 7% |
| Publication | Headline | Date | Hand Tuned Sentiment |
Textblob Sentiment |
Llama 3.1b LLM |
|---|---|---|---|---|---|
| Express | White van men are going green as they switch to electric vehicles | 2023-09-20 | Positive | Neutral | Negative |
| The Daily Mail | BMWs new electric i5 offers supercar performance in an executive saloon package - but is it better than a Tesla | 2024-01-08 | Negative | Positive | Neutral |
| The Sun | Electric car sales plummet as relieved drivers opt for petrol and diesel motors | 2023-10-05 | Negative | Positive | Neutral |
| The Sun | I took my EV on a 280-mile trip across the UK... it was a disaster | 2023-09-19 | Negative | Positive | Neutral |
| The Sun | Major rental company makes U-turn on EV rollout over important issue | 2023-10-31 | Negative | Positive | Neutral |
| BBC | Car company Nissan promises to make electric vehicles in Sunderland | 2023-11-23 | Positive | Negative | Neutral |
| The Telegraph | BT to turn street cabinets into electric car chargers | 2024-01-08 | Positive | Negative | Neutral |
| The Daily Mail | What drains an electric car battery most From air-con to smartphone charging - how much range is wiped-out when using these nine common features | 2023-08-02 | Negative | Positive | Positive |
| The Sun | Urgent warning over EVs after batteries suddenly explode into flames | 2023-10-11 | Negative | Positive | Neutral |
While I have taken steps to reduce personal biases during classification, it is possible that these biases could affect the accuracy of the findings. As such, the results presented in this report should be interpreted with caution and considered within the context of the broader study.