Error attempting to claim book from newsletter

[lucymhdavies]

~ $ python script/spider.py --config config/prod.cfg --notify ifttt --claimOnly

                      __   __              __                                   __
    ____  ____ ______/ /__/ /_____  __  __/ /_        ______________ __      __/ /__  _____
   / __ \/ __ `/ ___/ //_/ __/ __ \/ / / / __ \______/ ___/ ___/ __ `/ | /| / / / _ \/ ___/
  / /_/ / /_/ / /__/ ,< / /_/ /_/ / /_/ / /_/ /_____/ /__/ /  / /_/ /| |/ |/ / /  __/ /
 / .___/\__,_/\___/_/|_|\__/ .___/\__,_/_.___/      \___/_/   \__,_/ |__/|__/_/\___/_/
/_/                       /_/

Download FREE eBook every day from www.packtpub.com
@see github.com/niqdev/packtpub-crawler

[*] 2017-01-31 10:30 - fetching today's eBooks
[*] configuration file: /app/config/prod.cfg
[*] getting daily free eBook
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
[+] book successfully claimed
[+] notification sent to IFTTT
[*] getting free eBook from newsletter
[*] fetching url... 200 | https://www.packtpub.com/packt/free-ebook/practical-data-analysis
[-] <type 'exceptions.IndexError'> list index out of range | spider.py@123
Traceback (most recent call last):
  File "script/spider.py", line 123, in main
    packtpub.runNewsletter(currentNewsletterUrl)
  File "/app/script/packtpub.py", line 160, in runNewsletter
    self.__parseNewsletterBookInfo(soup)
  File "/app/script/packtpub.py", line 98, in __parseNewsletterBookInfo
    title = urlWithTitle.split('/')[4].replace('-', ' ').title()
IndexError: list index out of range
[+] error notification sent to IFTTT
[*] done
~ $

It has successfully claimed the book from the newsletter already, but on subsequent days I'm getting the above error.

And it sends an IFTTT notification for the second one :(

[lucymhdavies]

Updated to the latest code from master of this repo. Issue still present.

~ $ python script/spider.py --config config/prod.cfg --claimOnly

                      __   __              __                                   __
    ____  ____ ______/ /__/ /_____  __  __/ /_        ______________ __      __/ /__  _____
   / __ \/ __ `/ ___/ //_/ __/ __ \/ / / / __ \______/ ___/ ___/ __ `/ | /| / / / _ \/ ___/
  / /_/ / /_/ / /__/ ,< / /_/ /_/ / /_/ / /_/ /_____/ /__/ /  / /_/ /| |/ |/ / /  __/ /
 / .___/\__,_/\___/_/|_|\__/ .___/\__,_/_.___/      \___/_/   \__,_/ |__/|__/_/\___/_/
/_/                       /_/

Download FREE eBook every day from www.packtpub.com
@see github.com/niqdev/packtpub-crawler

[*] 2017-01-31 10:36 - fetching today's eBooks
[*] configuration file: /app/config/prod.cfg
[*] getting daily free eBook
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
[+] book successfully claimed
[*] getting free eBook from newsletter
[*] fetching url... 200 | https://www.packtpub.com/packt/free-ebook/practical-data-analysis
[-] <type 'exceptions.IndexError'> list index out of range | spider.py@123
Traceback (most recent call last):
  File "script/spider.py", line 123, in main
    packtpub.runNewsletter(currentNewsletterUrl)
  File "/app/script/packtpub.py", line 160, in runNewsletter
    self.__parseNewsletterBookInfo(soup)
  File "/app/script/packtpub.py", line 98, in __parseNewsletterBookInfo
    title = urlWithTitle.split('/')[4].replace('-', ' ').title()
IndexError: list index out of range
[*] done

[juli-dot-md]

Can confirm, I'll look into it. For a quickfix, create a file named "lastNewsletterUrl" containing "https://www.packtpub.com/packt/free-ebook/practical-data-analysis" in the config folder. This should stop the errors for now.

[juli-dot-md]

Fixed in #48

[lucymhdavies]

nice. thanks 👍

[trancen]

Getting this error now after updating : on Version - 2.2.4

[+] new download: /home/david/packtpub-crawler//home/david/packtpub-crawler/ebooks/extras/MySQL_for_Python.zip
[*] getting free eBook from newsletter
[*] fetching url... 200 | https://www.packtpub.com/packt/free-ebook/practical-data-analysis
[-] <type 'exceptions.IndexError'> list index out of range | spider.py@123
Traceback (most recent call last):
  File "script/spider.py", line 123, in main
    packtpub.runNewsletter(currentNewsletterUrl)
  File "/home/david/packtpub-crawler/script/packtpub.py", line 160, in runNewsletter
    self.__parseNewsletterBookInfo(soup)
  File "/home/david/packtpub-crawler/script/packtpub.py", line 98, in __parseNewsletterBookInfo
    title = urlWithTitle.split('/')[4].replace('-', ' ').title()
IndexError: list index out of range
[*] done

[juli-dot-md]

Yes, this is the same error. The fix should be merged tonight hopefully or you can change the line yourself.

[trancen]

Sorry just noticed the comment above about created the "lastNewsletterUrl" that worked.

[niqdev]

Thanks @juzim , the problem above is fixed, but actually there is a bug, see the log below (I've hidden some variables/paths)

• upload 2 times the ebook from the newsletter with 2 different name (but the file is the same)
• I received 2 email, the first one is correct, the second again contains 2 link with 2 pdf but are the same

python script/spider.py -c config/prod.cfg -u drive -s firebase -n gmail


                      __   __              __                                   __
    ____  ____ ______/ /__/ /_____  __  __/ /_        ______________ __      __/ /__  _____
   / __ \/ __ `/ ___/ //_/ __/ __ \/ / / / __ \______/ ___/ ___/ __ `/ | /| / / / _ \/ ___/
  / /_/ / /_/ / /__/ ,< / /_/ /_/ / /_/ / /_/ /_____/ /__/ /  / /_/ /| |/ |/ / /  __/ /
 / .___/\__,_/\___/_/|_|\__/ .___/\__,_/_.___/      \___/_/   \__,_/ |__/|__/_/\___/_/
/_/                       /_/

Download FREE eBook every day from www.packtpub.com
@see github.com/niqdev/packtpub-crawler
        
[*] 2017-01-31 18:45 - fetching today's eBooks
[*] configuration file: XXX/github/packtpub-crawler/config/prod.cfg
[*] getting daily free eBook
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
[+] book successfully claimed
[-] downloading file from url: https://www.packtpub.com/ebook_download/12890/pdf
[################################] 9985/9985 - 00:00:01
[+] new download: XXX/packtpub-crawler/ebooks/MySQL_for_Python.pdf
[+] new file upload on Drive:
[+] uploading file...
-[+] updating file permissions...
-       [path] XXX/packtpub-crawler/ebooks/MySQL_for_Python.pdf
	[download_url] UUU
	[name] MySQL_for_Python.pdf
	[mime_type] application/pdf
	[id] III
[+] Stored on firebase: KKK
[+] notified to: ['aaa', 'bbb']
[*] getting free eBook from newsletter
[*] fetching url... 200 | https://www.packtpub.com/packt/free-ebook/practical-data-analysis
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
[+] book successfully claimed
[-] downloading file from url: https://www.packtpub.com/ebook_download/12890/pdf
[################################] 9985/9985 - 00:00:01
[+] new download: XXX/packtpub-crawler/ebooks/Practical_Data_Analysis.pdf
[+] new file upload on Drive:
[+] uploading file...
|[+] updating file permissions...
-       [path] XXX/packtpub-crawler/ebooks/MySQL_for_Python.pdf
	[download_url] https://drive.google.com/uc?id=ZZZ
	[name] MySQL_for_Python.pdf
	[mime_type] application/pdf
	[id] LLL
[+] new file upload on Drive:
[+] uploading file...
|[+] updating file permissions...
\       [path] XXX/packtpub-crawler/ebooks/Practical_Data_Analysis.pdf
	[download_url] https://drive.google.com/uc?id=DDD
	[name] Practical_Data_Analysis.pdf
	[mime_type] application/pdf
	[id] YYY
[+] Stored on firebase: WWW
[+] notified to: ['aaa', 'bbb']
[*] done

[niqdev]

Actually I started the script again and it seems that the 3 book are identical and the daily ebook is ignored and I have 3 copy of the same book (newsletter). Moreover I checked on firebase and the data uploaded are mixed e.g. different filename but same author (of the newsletter).

[juli-dot-md]

I think we have to reset the Packtpub instance before handling the newsletter but then this would have been broken for weeks. Could you check?

Also, could you comment out spider.py:22 so we can see the contents of each handled packtpub.info?

[niqdev]

Yep,
so first log (see the author is wrong) and noticed now paths is empty

...
[*] getting daily free eBook
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
{
  "url_source_code": "https://www.packtpub.com/code_download/12891", 
  "paths": [], 
  "description": "MySQL for Python is the essential ingredient for building productive and feature-rich Python applications as it provides powerful database support and will also take the burden off your webserver. This eBook shows how to boost the productivity and maintainability of your Python apps by integrating them with the MySQL database server. It will take you from installing MySQL for Python on your platform of choice all the way through to database automation and administration. Every chapter is illustrated with a practical project that you can use during your own app development process. This eBook is free for today only so don\u2019t miss out!", 
  "title": "MySQL for Python", 
  "author": "Hector Cuesta", 
  "filename": "MySQL_for_Python", 
  "book_id": "12890", 
  "url_claim": "https://www.packtpub.com/freelearning-claim/5286/21478", 
  "url_image": "https://dz13w8afd47il.cloudfront.net/sites/default/files/imagecache/dotd_main_image/0189OS_MockupCover_0.jpg"
}
[+] book successfully claimed
...

and second log

...
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
{
  "url_source_code": "https://www.packtpub.com/code_download/12891", 
  "paths": [
    "XXX/packtpub-crawler/ebooks/MySQL_for_Python.pdf"
  ], 
  "description": "Get started in data analysis with this free 360 page eBook guide\nFor small businesses, analyzing the information contained in their data using open source technology could be game-changing. All you need is some basic programming and mathematical skills to do just that. This free data analysis eBook is designed to give you the knowledge you need to start succeeding in data analysis. Discover the tools, techniques and algorithms you need to transform your data into insight.\n\nVisualize your data to find trends and correlations\nBuild your own image similarity search engine\nLearn how to forecast numerical values from time series data\nCreate an interactive visualization for your social media graph", 
  "title": "Practical Data Analysis", 
  "author": "Hector Cuesta", 
  "filename": "Practical_Data_Analysis", 
  "book_id": "12890", 
  "url_claim": "https://www.packtpub.com/promo/claim/12891/27564", 
  "url_image": "https://d1ldz4te4covpm.cloudfront.net/sites/default/files/B02731_Practical Data Analysis.jpg"
}
[+] book successfully claimed
...

[niqdev]

About the first question, we probably have to reset everything before a new claim, but is this the first newsletter since they reactiveted the free ebook? The other books seems correct, I checked also the logs on heroku

[niqdev]

Also the field url_source_code is wrong

[juli-dot-md]

Can you add packtpub = Packtpub(config, args.dev) to spider.py:123 and see if it fixes it? Sorry, but somehow the tests on my machine are broken...

[juli-dot-md]

Did you by any chance delete the lastNewsletterUrl file? Because if the script tries to grab an already claimed newsletter book from the archive, it won't find it on the top position and overwrite the data for whatever book is currently on there.

We have to either check if the book was already claimed (would be a nice feature anyways) or find the book in the list by name/id/etc.

I'll try to look into it tomorrow but it shouldn't happen again unless you delete the file.