Skip to content

This pull request fixes issues with submission format unmatching in 0.3.0 and 0.4.2#26

Open
thehir0 wants to merge 3 commits into
ai-forever:update/new_harness_codebasefrom
thehir0:update/new_harness_codebase
Open

This pull request fixes issues with submission format unmatching in 0.3.0 and 0.4.2#26
thehir0 wants to merge 3 commits into
ai-forever:update/new_harness_codebasefrom
thehir0:update/new_harness_codebase

Conversation

@thehir0

@thehir0 thehir0 commented Jul 10, 2024

Copy link
Copy Markdown
Contributor

RuTie in 0.3.0:

{
    "data": {
        "test": [
            [
                {
                    "outputs": "1",
                    "meta": {
                        "dialog_id": 0,
                        "question_id": 0
                    }
                },

RuTie in 0.4.2

{
    "data": {
        "test": [
            [
                {
                    "outputs": {
                        "outputs": "1",
                        "meta": {
                            "id": 0
                        }
                    },
                    "meta": {
                        "dialog_id": 0,
                        "question_id": 0
                    }
                },

RuHumanEval in 0.3.0

{
    "data": {
        "test": [
            {
                "outputs": [
                    [
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null
                    ],
                    [
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null,
                        null
                    ],

RuHumanEval in 0.4.2(Has double squared brackets in outputs)

{
    "data": {
        "test": [
            {
                "outputs": [
                    [
                        [
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None"
                        ],
                        [
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None",
                            "None"
                        ],
                        [

@thehir0

thehir0 commented Jul 10, 2024

Copy link
Copy Markdown
Contributor Author

Without this fix: i get 0.0 ruTie and 0/1/1 ruHumanEval on submission page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant