Sentinel

I built Sentinel to look at the information lost when financial news, social discussion, and official filings get reduced to a single sentiment score, using a fine-tuned LLaMA 3 model alongside two metrics I designed to measure disagreement: SDI and MASLD.

Research context

The thesis treats disagreement as useful information rather than noise. I developed the Sentiment Disagreement Index and its aspect-level extension, MASLD, to identify where sources diverge on revenue, management, risk, and market perception.

Research Scope

Research and modelling

  • A five-phase research pipeline: FinBERT baseline, multi-source data collection, LLaMA fine-tuning, the SDI metric, then the MASLD framework
  • LLaMA 3 8B, fine-tuned with QLoRA for financial sentiment classification
  • The Sentiment Disagreement Index (SDI), a credibility-weighted pairwise distance across source types
  • SDI extended to the aspect level, as Multi-Source Aspect-Level Sentiment Divergence (MASLD)

Data and evaluation

  • Multi-source data collection: news via the Guardian API, Reddit via the Arctic Shift API, SEC filings via EDGAR
  • A Tesla Q1 2024 earnings case study, used to validate divergence findings against a real market event
  • Benchmarking against Financial PhraseBank across 75%, 66%, and 50% analyst-agreement tiers

Completed Work

FinBERT sentiment baseline

A baseline financial-sentiment classifier used to establish a comparison point before the LLaMA fine-tune.

Built with
FinBERT, PyTorch
Contribution
Trained and evaluated the baseline as Phase 1 of the thesis pipeline.
Outcome
97.17% accuracy on the baseline task.

LLaMA 3 8B fine-tune (QLoRA)

A parameter-efficient fine-tune of LLaMA 3 8B on financial text, targeting sentiment classification accuracy competitive with much larger general-purpose models.

Built with
LLaMA 3 8B, QLoRA, Hugging Face Transformers, Kaggle T4 GPUs
Contribution
Ran the fine-tuning pipeline end to end on free-tier dual T4 compute.
Outcome
96% macro F1 on Financial PhraseBank, ahead of GPT-4's reported 87% on the same benchmark.

Sentiment Disagreement Index (SDI) and MASLD

Two novel metrics: SDI measures credibility-weighted sentiment disagreement across source types (SEC 0.9, news 0.7, social 0.4); MASLD extends the same idea to individual aspects within a document for more granular analysis.

Built with
Python, Statistical modelling
Contribution
Designed both metrics and validated them against the Tesla Q1 2024 earnings case study.
Outcome
A reusable framework for surfacing where sources disagree, not just an averaged sentiment score.

Technical Stack

Technical skills

LLM fine-tuning (QLoRA), Financial NLP, Statistical modelling, Research methodology

Tools and platforms

PyTorch, Hugging Face Transformers, LLaMA 3, FinBERT, Kaggle

Data sources

Guardian API, Arctic Shift API (Reddit), SEC EDGAR

Research Pipeline

  1. Establishing the FinBERT baseline

    A FinBERT sentiment baseline provides the reference point for every later result.

  2. Collecting multi-source data

    The pipeline collects news, Reddit, and SEC filing data through the Guardian, Arctic Shift, and EDGAR APIs.

  3. Fine-tuning LLaMA

    LLaMA 3 8B is fine-tuned with QLoRA on financial text for sentiment classification.

  4. Defining the disagreement index

    SDI measures credibility-weighted pairwise sentiment distance across source types.

  5. Extending the method to MASLD

    MASLD extends disagreement measurement to the aspect level and is validated against the Tesla Q1 2024 earnings case study.

Problems & Handling

Fine-tuning an 8B-parameter model meant working within Kaggle's free-tier dual T4 GPUs, with no paid infrastructure to fall back on.

Full fine-tuning at that scale wasn't feasible within the GPU memory and time limits of the free tier.

QLoRA made it workable: quantizing the base model and training low-rank adapters instead of all 8B parameters.

96% macro F1 on Financial PhraseBank, on free-tier hardware, ahead of GPT-4's reported 87%.

News, social media, and SEC filings routinely disagreed on sentiment, but most sentiment tools collapse that into one averaged number.

Averaging throws away the signal that disagreement itself carries, especially around volatile events like earnings calls.

The Sentiment Disagreement Index and its aspect-level extension, MASLD, were built to measure and expose disagreement directly, weighted by source credibility, instead of averaging it away.

A framework validated against a real event, Tesla's Q1 2024 earnings, that surfaces cross-source disagreement instead of hiding it.

Summary

FinBERT baseline accuracy
97.17%
LLaMA 3 8B macro F1 (Financial PhraseBank)
96%, vs. GPT-4's reported 87%
Novel contributions
SDI, MASLD
Publication status
IEEEtran thesis document in progress; conference paper in parallel
  • Validated against a real market event, Tesla's Q1 2024 earnings, rather than only against static benchmark data.
  • Parts of the codebase and findings are planned for open-sourcing around submission.

Looking back

  • QLoRA was the difference between this being possible on free Kaggle GPUs and not being possible at all. An 8B model fine-tune on that little compute still beat GPT-4's reported number on the same benchmark.
  • Building SDI and MASLD changed how I think about sources disagreeing with each other. It's not noise you average away, it's actually a signal worth measuring on its own.