google chrome - Get request with Python requests returning different page than seen in browser -


i have been trying request on youtube video page in order read simple information off of page. have done many times before, , quite easy reverse engineer request of google chrome's developer tools.

to demonstrate, here screen shot of request when reload youtube video in fresh incognito window (to prevent cookies being sent) seen developer menu: chrome screenshot

every time close window , reload page recieve identical html (apart authorization keys , like), bottom of can seen here: another chrome screenshot

first tried recreating request using header-less requests in python:

import requests  sesh = requests.session() print sesh.get("https://www.youtube.com/watch?v=5ea8ivrqwn8").content

this returns different page still contains of data present on page chrome not of it. next tried including headers saw in chrome request using following code:

import requests  sesh = requests.session() headers = {                 "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",                 "accept-encoding": "gzip, deflate, br",                 "accept-language":"en-us,en;q=0.8",                 "upgrade-insecure-requests": "1",                 "user-agent": "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, gecko) chrome/60.0.3112.113 safari/537.36"}  print sesh.get("https://www.youtube.com/watch?v=5ea8ivrqwn8", headers = headers).content

however strangely returns seemingly random quick paragraph of unicode characters in varying lengths, around 10 characters long, closer 50. couldn't think of other ways make closer request seeing chrome. tried fiddling couple of hours doing things running request multiple times in same session , messing headers bit, no avail.

finally out of desperation tried dropping except user agent, using following code:

import requests  sesh = requests.session() headers = {"user-agent": "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, gecko) chrome/60.0.3112.113 safari/537.36"} print sesh.get("https://www.youtube.com/watch?v=5ea8ivrqwn8", headers = headers).content

and got me page wanted.

however left unsatisfied knowledge somehow replicating seeing in chrome didn't work. missing second attempt?


Comments

Post a Comment

Popular posts from this blog

resizing Telegram inline keyboard -

telegram's inline keyboard great feature lots of different use cases. inline buttons added list of items this: inline_keyboard = [[inlinekeyboardbutton(text="button", callback_data="button"), inlinekeyboardbutton(text="reset",callback_data="reset")]] inline_keyboard_markup = inlinekeyboardmarkup(inline_keyboard) update.message.reply_text("hi", reply_markup=inline_keyboard_markup) the above code adds 2 buttons each half width of chat screen. i know the normal keyboard button there resize_keyboard parameter somehow can used. my question is there way resize inline buttons ? example make full width or quarter width. unfortunately, can't now. :( you can suggest @botsupport , might add feature next version.
Read more

command line - How can a Python program background itself? -

consider following python command-line program. doesn't work because incomplete, think illustrates problem: #!/usr/bin/python3 import multiprocessing pool = multiprocessing.pool() task in expensive_iterator_with_user_interaction(): pool.call_async(expensive_computation, (task,)) pool.close() print("dear user, may press ctrl-z , bg script.") pool.join() thus, problem scripts needs user input while it's launching expensive parallel processes. there moment when user interaction not needed anymore. then, ask user background process shell actions. however, can achieved program itself? (side note: core of problem entanglement of user input firing “expensive computations”. therefore, cannot separate both in 2 scripts, , letting 1 spawning other daemon.) the command screen may you. use screen -l create new session , run application. after inputting data, can press " ctrl-a " , press " d " on keyboard detach session. now
Read more

php - "cURL error 28: Resolving timed out" on Wordpress on Azure App Service on Linux -

Image
my wordpress installation on azure app service on linux using custom docker container has slow response times. pages take around 20-40 seconds load. i have troubleshooting plugin installed indicates problem "curl error 28: resolving timed out after n milliseconds" when making requests following urls https://api.wordpress.org/core/version-check/1.7/ http://api.wordpress.org/core/version-check/1.7/ https://api.wordpress.org/plugins/update-check/1.1/ http://api.wordpress.org/plugins/update-check/1.1/ https://api.wordpress.org/themes/update-check/1.1/ http://api.wordpress.org/themes/update-check/1.1/ curl works fine on scm-site command line. example works ok. curl -x post http://api.wordpress.org/core/version-check/1.7/ edit if ssh container , run php code, works fine. <?php $url = 'http://api.wordpress.org/core/version-check/1.7/'; $fields = array( 'version' => urlencode('4.8.1'), 'php' => urlencode(
Read more