Hi there,
First off, I just want to say huge thanks for building this- it helped me build a nice scraper to track/store my electricity usage which is something I've been putting off for a very long time.
While trying to get this library to work I noticed that I would successfully fetch reading data for only about 25% of my attempts. The other 75% would get caught up on authentication issues or some element not showing up as it should. I had a feeling these issues were due to various race conditions associated with trying to scrape JS-heavy webpage and wanted to do some refactoring, taking some lessons from how Cypress approaches this kind of work. Namely, I restructured the scraper to watch for and respond to elements appearing/disappearing on the page rather than waiting arbitrary amounts of time and hoping requests finished.
Here's the code, which is built pretty specifically for the context where I'm using it, but you can see the general ideas:
async def fetch_element(page, selector, max_tries=10):
tries = 0
el = None
while el == None and tries < max_tries:
el = await page.querySelector(selector)
await page.waitFor(1000)
return el
async def fetch_readings():
browser_launch_config = {
'defaultViewport': {'width': 1920, 'height': 1080},
'dumpio': False,
'args': ['--no-sandbox'],
}
browser = await pyppeteer.launch(browser_launch_config)
page = await browser.newPage()
await page.goto('https://coned.com/en/login', {'waitUntil' : 'domcontentloaded'})
element = await page.querySelector('#form-login-email')
logging.info('Authenticating...')
await page.type('#form-login-email', os.environ.get('CONED_EMAIL'))
await page.type('#form-login-password', os.environ.get('CONED_PASSWORD'))
await page.click('.submit-button'),
mfa_form = await fetch_element(page, '.js-login-new-device-form-selector:not(.hidden)')
if mfa_form is None:
logging.error('Never got MFA prompt. Aborting!')
return
logging.info('Entering MFA code...')
mfa = await fetch_element(page, '#form-login-mfa-code')
mfa_code = pyotp.TOTP(os.environ.get('CONED_MFA')).now()
await mfa.type(mfa_code)
await asyncio.gather(
page.waitForNavigation(),
page.click('.js-login-new-device-form .button'),
)
logging.info('Pausing for auth...')
await page.waitFor(5000)
logging.info('Fetching readings JSON...')
api_page = await browser.newPage()
account_id = os.environ.get('CONED_ACCOUNT_ID')
meter_no = os.environ.get('CONED_METER_NUMBER')
url = f"https://cned.opower.com/ei/edge/apis/cws-real-time-ami-v1/cws/cned/accounts/{account_id}/meters/{meter_no}/usage"
await api_page.goto(url)
data_elem = await api_page.querySelector('pre')
raw_data = await api_page.evaluate('(el) => el.textContent', data_elem)
data = json.loads(raw_data)
await browser.close()
logging.info('Done!')
return dataThe results have been promising so far- in my testing this method has been more reliable and faster than the current implementation since it doesn't have to wait as long for data. I figured I'd share it here in case you'd like to incorporate the changes into your library (or if you're open to a PR I can see if I can make the changes myself), or for others to use as a reference.
One thing I'd also like to add: I would recommend returning all of the reading results rather than just the latest one. AMI data can be lagged or be updated as time goes on (utilities are bad at computers), so if you're trying to scrape and store your meter's data you'll likely want to fetch the whole set of readings and insert/replace each interval in the database you're storing them in. Since running this scraper I've noticed that the latest reading is usually null for an hour or so before it starts getting populated with a kwh value.
Hope this is helpful!
Hi there,
First off, I just want to say huge thanks for building this- it helped me build a nice scraper to track/store my electricity usage which is something I've been putting off for a very long time.
While trying to get this library to work I noticed that I would successfully fetch reading data for only about 25% of my attempts. The other 75% would get caught up on authentication issues or some element not showing up as it should. I had a feeling these issues were due to various race conditions associated with trying to scrape JS-heavy webpage and wanted to do some refactoring, taking some lessons from how Cypress approaches this kind of work. Namely, I restructured the scraper to watch for and respond to elements appearing/disappearing on the page rather than waiting arbitrary amounts of time and hoping requests finished.
Here's the code, which is built pretty specifically for the context where I'm using it, but you can see the general ideas:
The results have been promising so far- in my testing this method has been more reliable and faster than the current implementation since it doesn't have to wait as long for data. I figured I'd share it here in case you'd like to incorporate the changes into your library (or if you're open to a PR I can see if I can make the changes myself), or for others to use as a reference.
One thing I'd also like to add: I would recommend returning all of the reading results rather than just the latest one. AMI data can be lagged or be updated as time goes on (utilities are bad at computers), so if you're trying to scrape and store your meter's data you'll likely want to fetch the whole set of readings and insert/replace each interval in the database you're storing them in. Since running this scraper I've noticed that the latest reading is usually
nullfor an hour or so before it starts getting populated with a kwh value.Hope this is helpful!