[Bug] Webtoons Plugin broken: "Search failed. Error: Failed to get chapter list." (and I think I know why!)

[GlassedSilver]

Plugin
webtoons

Describe the bug
The plugin can connect to Webtoons, however getting the chapter list fails as described in the issue title.
I think this is due to missing cookies and user agent.

FMD2 has had the same problem for a while. It can get a description of the toon, but if you try to pull the chapter list it just stalls.

See:
dazedcat19/FMD2#187 (comment)

The workaround is described there as well. I THINK that this applies here as well.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Download'
2. Choose 'Webtoons'
3. Search for a valid webtoon title
4. See error

Expected behavior
Not to get an error and see results :P

Screenshots
not needed

Mango version
Latest release on docker in unRAID as of now.
So should be 0.23.0

Additional context
n/a I think

[hkalexling]

Are you still experiencing the issue? I couldn't reproduce it. I can search with the plugin just fine.

[GlassedSilver]

Yup I do! In the issue I linked the issue was not always reproducible as well, seems like that's a bit of a nature to this specific issue, that it's not always there.

[YaBoiMax]

I think I know why this issue exists, I just did some sniffing through the various return values until I found that in the section

try {
          var resp = mango.get(BASE_URL + LIST_ENDPOINT + mangaID);
          var urlLocation = resp.headers.Location;
  } catch (error) {
          mango.raise("Could not get webtoon page.");
  }

The line resp.headers.Location will return /en/gdpr/ageGate which I don't think is the expected return value and instead redirects you to the age verification page

[hkalexling]

@YaBoiMax Thanks for investigating the issue! I don't use the webtoons.com myself, so I am not sure how their age verification works. Also @GlassedSilver was searching for "hello", which should match to this. I don't see how this would trigger the age verification.

[hkalexling]

I checked the cookies of the site and I noticed the cookie needCOPPA. For me it's set to false, but I guess depending on the country you are in it might be true, which might trigger the age verification.

[cesarblancg]

Today I installed mango and I had the same error. In europe, the GDPR it's a nightmare to scrap.

I researched the cookies that webtoon needs and it seems that changing GDPR=false is enough.

I tried to add these headers:

var headers = {
'Cookie': 'needGDPR=false',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36 Edg/101.0.1210.32',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'es,es-ES;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6'
};

but it seems that mango.get() it's not following the 302. Location seems good but the body is empty:

Do you have any idea about it?

[hkalexling]

@cesarblancg Thanks for testing this! Yeah Mango uses the HTTP::Client from Crystal which doesn't follow redirects (crystal-lang/crystal#2721). But we should be able to explicitly handle the 302 status code and send another GET request in the plugin code.

[KiwiHuman]

I am also having issues with the webtoon plugin. looking at the output of the terminal mango is running in

[ERROR] 2023/06/22 17:55:22 | Hostname lookup for ac.webtoons.com failed: No address found

after changing SEARCH_URL to www.webtoons.com instead to see if that fixed the issue I now get the error

[ERROR] 2023/06/22 18:02:17 | An error occured while searching.

if I try to search for something.

[YaBoiMax]

I don't know why your SEARCH_URL is/was set to ac.webtoons.com as that isn't a valid subdomain of webtoons.
Now it's been a long time since I last tried to use Mango but I think I got the same error message (An error occured while searching.) as you. Are you located anywhere inside of the EU?

[KiwiHuman]

I am in England so kind of (Brexit made a mess of anything to do with EU), the GDPR issues mentioned earlier on this issue could be effecting me.

[YaBoiMax]

I just quickly spun up mango on a Linux Container and have been looking around trying to find answers.
I think the webtoons plugin is completely nonfunctional at the moment due to the fact that webtoons seems to have changed how you can search things on the site.

In the script this part fails.

try {
	var searchResp = mango.get(SEARCH_URL + encodeURI(query) + SEARCH_PARAMS).body;
	var search = JSON.parse(searchResp);
} catch (error) {
	mango.raise("An error occured while searching.");
}

Due to the fact that the ac.webtoons.com domain no longer exists you cannot search through that. You also need a return value that is JSON or resort to webscraping. Because looking at how webtoons lets you search on their website right now the SEARCH_URL could be https://www.webtoons.com/en/search?searchType=CHALLENGE&keyword=$QUERY but that returns plain html and only a single page, for mutliple pages you have to add &page=$PAGENUMBER to get the rest.