Hi,
we have some HTML that contains links like these:
<a href="http://link.springer.com/article/10.1007%2Fs12064-012-0164-1">foo</a>
This link is found by rawler, but it is re-encoded to http://link.springer.com/article/10.1007%252Fs12064-012-0164-1. Note: %2 before vs. %252 after.
The check is then performed using the incorrect URL containing %252. Of course, it yields a 404 and reports the link as broken, while browsers and the W3C validator consider it valid.
I traced it down to this line: https://github.com/oscardelben/rawler/blob/master/lib/rawler/crawler.rb#L44
Running Ruby 2.3.1.
Hi,
we have some HTML that contains links like these:
This link is found by rawler, but it is re-encoded to
http://link.springer.com/article/10.1007%252Fs12064-012-0164-1. Note:%2before vs.%252after.The check is then performed using the incorrect URL containing
%252. Of course, it yields a 404 and reports the link as broken, while browsers and the W3C validator consider it valid.I traced it down to this line: https://github.com/oscardelben/rawler/blob/master/lib/rawler/crawler.rb#L44
Running Ruby 2.3.1.