✨FEATURE: Add emotion timeline output to Sentiment API

[jvJUCA]

Context

Currently, the Sentiment API returns only the overall average of emotions across the entire video, in the following format:

{
  "emotions": {
    "Angry": 15.38,
    "Disgusted": 7.69,
    "Fearful": 0.0,
    "Happy": 0.0,
    "Neutral": 53.85,
    "Sad": 23.08,
    "Surprised": 0.0
  }
}

This is not enough for the frontend, which needs to build an emotional timeline throughout the video.

---

Problem

• There is no information about when each emotion occurs
• It is not possible to render charts or timelines
• A global average loses important temporal context

---

Proposal

In addition to the overall average, the API should return a time-based array segmented by fixed intervals.

---

Expected payload

{
  "average_emotions": {
    "Angry": 15.38,
    "Disgusted": 7.69,
    "Fearful": 0.0,
    "Happy": 0.0,
    "Neutral": 53.85,
    "Sad": 23.08,
    "Surprised": 0.0
  },
  "timeline": [
    {
      "id": 1,
      "starting_time": "00:00",
      "ending_time": "00:10",
      "emotion": "SURPRISED",
      "value": 54
    },
    {
      "id": 2,
      "starting_time": "00:10",
      "ending_time": "00:20",
      "emotion": "NEUTRAL",
      "value": 62
    }
  ]
}

---

Requirements

• Configurable time intervals (e.g. 5s, 10s) or when change the dominant emotion
• Dominant emotion per interval
• value representing intensity/confidence (%)
• Maintain backward compatibility (overall average must still be returned)

---

Benefits

• Enables emotion timeline visualization on the frontend
• Provides richer context for UX analysis
• Data can be reused for charts and insights

---

Note

The frontend is already prepared to consume this format.

[Santosh69]

Hi @jvJUCA, I can pick this up. I'm looking at the frontend issue (#1227), handling this backend part would help me ensure the API returns exactly what the UI needs for the timeline. Happy to start immediately if you approved

Thanks!

[jvJUCA]

Hi @Santosh69 , your contribution will be appreciated. I’ll assign you to the issue!

[gauravk16in]

@jvJUCA can I work on this ?

[Santosh69]

Hi @gauravk16in, thanks for checking out
I’m actively working on this issue and testing if the implementation is working correctly, while making sure it integrates cleanly with the main codebase/dependencies.
To avoid duplicate work and extra review overhead, I’ll continue with this implementation for now.
We can connect on Discord, if I need review or help

Appreciate the understanding!

[gauravk16in]

@Santosh69 It's fine! You do it I will not bother.

[Nirvanjha2004]

Hi @jvJUCA ! Greetings !
I have read the research paper EmoBERTa Speaker-Aware Emotion Recognition in Conversation with RoBERTa , there it was mentioned that we have to feed both the past and future sentences as well to get best output. But in our current implementation, we were only feeding the current sentence to get the output. In my implementation i have passed both the future , past , current sentences and timeline as well for better prediction of emotions with the expected output as mentioned in issue. Kindly Review and suggest changes if any !
Thanks for your review !

[Santosh69]

Hi @Nirvanjha2004, thanks for your enthusiasm and interest in contributing! 👋

I noticed you opened this PR for issue #14, which is currently assigned to me. I've been actively working on this implementation.

After reviewing your PR, I noticed it only includes the service-layer logic but is missing several critical components required by the issue:

• No API endpoint - The analyze_timeline() method isn't exposed via any route, so the frontend can't actually consume it
• Time format mismatch - The issue specifies "starting_time": "00:00" format (string), but this returns floats
• Missing configurable intervals - The issue requires configurable time intervals (5s, 10s, etc.)
• No integration with the pipeline - This needs to work with the existing audio-transcription-sentiment flow
• No tests - We need both unit and integration tests

Additionally:

• Frontend can't use it : service method can't be called by the frontend at all.
• Missing dominant emotion aggregation: Issue requires "Dominant emotion per interval", but the PR returns each Whisper chunk individually instead of aggregating to find the dominant emotion per time window (Please have look at the description ).

I'm making good progress on the complete implementation that addresses all these requirements alignedly and integrates properly with the frontend expectations (issue #1227)(Although i focused on understanding the relevant code in both repos), both PRs will be opened at once in about a day or less.

Would you mind closing this PR to avoid duplicate work and review overhead? Please Reach out to the Assignee about the progress before, instead of raising the PR, If you'd like to contribute, maybe we could coordinate on Discord first, there are other open issues where help would be appreciated!

Thanks for understanding 🙏

[Nirvanjha2004]

Hi @Santosh69 , thanks for the review!

I completely understand. Since the issue was marked 'Urgent' and had been inactive for over two weeks, I mistakenly assumed it was stalled and jumped in to get the ball rolling. I wasn't aware you were actively working on it locally.

You are absolutely right that my PR is missing the API and aggregation logic (I focused purely on the service-layer accuracy using EmoBERTa context). Since you have the complete implementation underway, I will close this PR to avoid conflicts.

Quick note: While working on this, I implemented the 'Sandwich' context logic (feeding Past+Current+Future chunks), which significantly improves sentiment accuracy compared to isolated chunks. Once your PR is merged, I’d love to open a separate enhancement issue to integrate that logic if you think it fits.

Thanks for clarifying!

[Swastik19Nit]

@jvJUCA I see the current codebase has sentiment analysis as POS/NEG/NEU but the issue mentions emotions (Angry, Disgusted, Feraful , Happy,, Neutral ,Sad, Surpised). Is there a separate emotion analysis endpoint/API that I have not found yet or does this feature need to be built from scratch??

Also for the timeline feature, should the time intervals be configurable via API request parameters ( eg interval_seconds = 10) or should they be fixed in the configuration file