U gE@sdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZmZmZmZddlmZddlmZmZmZdd lmZdd lmZdd lmZmZee Z!e"d d dZ#ddZ$ddZ%e"e&dddZ'e"e"e"fdddZ(e"e"dddZ)e"e"dddZ*e"e"dddZ+e"e"e"dd d!Z,e"e"e"d"d#d$Z-ee"d%d&d'Z.e"e"d(d)d*Z/e"e e"e e"fd+d,d-Z0e"e d(d.d/Z1e"e d(d0d1Z2e"e"d(d2d3Z3e e e"dd4d5d6Z4e dd7d8d9Z5e"e"dd:d;Z6e"e"ddd?d@Z8dS)AzBThis module contains all non-cipher related data extraction logic.N) OrderedDict)datetime)AnyDictListOptionalTuple)parse_qsquote urlencodeurlparse)Cipher)HTMLParseErrorLiveStreamErrorRegexMatchError regex_search)YouTubeMetadata)parse_for_objectparse_for_all_objects watch_htmlcCs6ztd|dd}Wntk r(YdSXt|dS)zExtract publish date :param str watch_html: The html contents of the watch page. :rtype: str :returns: Publish date of the video. z;(?<=itemprop=\"datePublished\" content=\")\d{4}-\d{2}-\d{2}rgroupNz%Y-%m-%d)rrrstrptime)rresultr2/tmp/pip-unpacked-wheel-1a9f0fi6/pytube/extract.py publish_dates rcCs"dg}|D]}||kr dSq dS)zCheck if live stream recording is available. :param str watch_html: The html contents of the watch page. :rtype: bool :returns: Whether or not the content is private. z,This live stream recording is not available.FTr)rZunavailable_stringsstringrrrrecording_available&s r cCs&dddg}|D]}||krdSqdS)zCheck if content is private. :param str watch_html: The html contents of the watch page. :rtype: bool :returns: Whether or not the content is private. zFThis is a private video. Please sign in to verify that you may see it.z"simpleText":"Private video"zThis video is private.TFr)rZprivate_stringsrrrr is_private8s r!)rreturncCs.ztd|ddWntk r(YdSXdS)zCheck if content is age restricted. :param str watch_html: The html contents of the watch page. :rtype: bool :returns: Whether or not the content is age restricted. zog:restrictions:agerrFT)rrrrrris_age_restrictedLs r#cCsdt|}|di}d|kr dSd|krZd|krB|d|dgfSd|krZ|d|dfSddgfS)aReturn the playability status and status explanation of a video. For example, a video may have a status of LOGIN_REQUIRED, and an explanation of "This is a private video. Please sign in to verify that you may see it." This explanation is what gets incorporated into the media player overlay. :param str watch_html: The html contents of the watch page. :rtype: bool :returns: Playability status and reason of the video. playabilityStatusliveStreamability)Z LIVE_STREAMzVideo is a live stream.statusreasonmessagesN)initial_player_responseget)rZplayer_responseZ status_dictrrrplayability_status\s r+)urlr"cCstd|ddS)arExtract the ``video_id`` from a YouTube url. This function supports the following patterns: - :samp:`https://youtube.com/watch?v={video_id}` - :samp:`https://youtube.com/embed/{video_id}` - :samp:`https://youtu.be/{video_id}` :param str url: A YouTube url containing a video id. :rtype: str :returns: YouTube video id. z(?:v=|\/)([0-9A-Za-z_-]{11}).*rr)r,rrrvideo_idvsr.cCstj|}t|jddS)aoExtract the ``playlist_id`` from a YouTube url. This function supports the following patterns: - :samp:`https://youtube.com/playlist?list={playlist_id}` - :samp:`https://youtube.com/watch?v={video_id}&list={playlist_id}` :param str url: A YouTube url containing a playlist id. :rtype: str :returns: YouTube playlist id. listr)urllibparser r query)r,parsedrrr playlist_ids r4cCsrddddg}|D]P}t|}||}|rtd||d}|d}d|d|Sqtd d d d S) aExtract the ``channel_name`` or ``channel_id`` from a YouTube url. This function supports the following patterns: - :samp:`https://youtube.com/c/{channel_name}/*` - :samp:`https://youtube.com/channel/{channel_id}/* - :samp:`https://youtube.com/u/{channel_name}/*` - :samp:`https://youtube.com/user/{channel_id}/* :param str url: A YouTube url containing a channel name. :rtype: str :returns: YouTube channel name. z(?:\/(c)\/([%\d\w_\-]+)(\/.*)?)z%(?:\/(channel)\/([%\w\d_\-]+)(\/.*)?)z(?:\/(u)\/([%\d\w_\-]+)(\/.*)?)z"(?:\/(user)\/([%\w\d_\-]+)(\/.*)?)"finished regex search, matched: %sr-/ channel_namepatternsZcallerpatternNrecompilesearchloggerdebugrr)r,r9r;regexfunction_matchZ uri_styleZuri_identifierrrrr8s"     r8)r. watch_urlr"cCs*td|fddt|fddddg}t|S)aConstruct the video_info url. :param str video_id: A YouTube video identifier. :param str watch_url: A YouTube watch url. :rtype: str :returns: :samp:`https://youtube.com/get_video_info` with necessary GET parameters. r.)Zpsdefaulteurl)Zhlen_UShtml51cZTVHTML5Zcverz 7.20201028)rr _video_info_url)r.rDparamsrrrvideo_info_urls   rP)r. embed_htmlr"cCs^ztd|dd}Wntk r*d}YnXd|}td|fd|fd|fd d d g}t|S) a<Construct the video_info url. :param str video_id: A YouTube video identifier. :param str embed_html: The html contents of the embed page (for age restricted videos). :rtype: str :returns: :samp:`https://youtube.com/get_video_info` with necessary GET parameters. z"sts"\s*:\s*(\d+)r-rz!https://youtube.googleapis.com/v/r.rFstsrHrKrM)rrrrN)r.rQrSrFrOrrrvideo_info_url_age_restricteds    rT)rOr"cCs dt|S)Nz'https://www.youtube.com/get_video_info?)r )rOrrrrNsrN)htmlr"c Cs>zt|dd}Wn ttfk r4t|}YnXd|S)zGet the base JavaScript url. Construct the base JavaScript url, which contains the decipher "transforms". :param str html: The html contents of the watch page. Zassetsjszhttps://youtube.com)get_ytplayer_configKeyErrorrget_ytplayer_js)rUZbase_jsrrrjs_urls rZ)mime_type_codecr"cCsLd}t|}||}|s(td|d|\}}|dd|dDfS)aParse the type data. Breaks up the data in the ``type`` key of the manifest, which contains the mime type and codecs serialized together, and splits them into separate elements. **Example**: mime_type_codec('audio/webm; codecs="opus"') -> ('audio/webm', ['opus']) :param str mime_type_codec: String containing mime type and codecs. :rtype: tuple :returns: The mime type and a list of codecs. z,(\w+\/\w+)\;\scodecs=\"([a-zA-Z-0-9.,\s]*)\"r[r:cSsg|] }|qSr)strip).0rLrrr $sz#mime_type_codec..,)r=r>r?rgroupssplit)r[r;rBresultsZ mime_typecodecsrrrr[ s    r[cCsVdg}|D]:}t|}||}|r td||d}|Sq tddddS)zGet the YouTube player base JavaScript path. :param str html The html contents of the watch page. :rtype: str :returns: Path to YouTube's base.js file. z'(/s/player/[\w\d]+/[\w\d_/.]+/base\.js)r5r-rYjs_url_patternsr:Nr<)rUrdr;rBrCZ yt_player_jsrrrrY's      rYc Cstdddg}|D]Z}zt||WStk rn}z&td|t|WYqW5d}~XYqXqdg}|D].}zt||WStk rYq|Yq|Xq|tddd dS) aGet the YouTube player configuration data from the watch html. Extract the ``ytplayer_config``, which is json data embedded within the watch html and serves as the primary source of obtaining the stream manifest data. :param str html: The html contents of the watch page. :rtype: str :returns: Substring of the html containing the encoded manifest data. zfinding initial function namezytplayer\.config\s*=\s*ytInitialPlayerResponse\s*=\s*zPattern failed: Nz,yt\.setConfig\(.*['\"]PLAYER_CONFIG['\"]:\s*rWz#config_patterns, setconfig_patternsr:)r@rArrr)rUZconfig_patternsr;eZsetconfig_patternsrrrrW@s,   rWc Csri}ddg}|D]@}z"t||}|D]}||q$Wqtk rNYqYqXqt|dkrb|StddddS)a;Get the entirety of the ytcfg object. This is built over multiple pieces, so we have to find all matches and combine the dicts together. :param str html: The html contents of the watch page. :rtype: str :returns: Substring of the html containing the encoded manifest data. z ytcfg\s=\sz ytcfg\.set\(r get_ytcfgZytcfg_pattenrsr:N)rupdaterlenr)rUZytcfgZytcfg_patternsr;Z found_objectsobjrrrrgns"    rg)stream_manifestvid_inforVr"c Cs8t|d}t|D]\}}z |d}Wn2tk rZ|did}|rVtdYnXd|ks|d|krd|ks|d |krtd q|j|dd }td |d t|} t t|j } dd| D} || d<d| krt | d} || } | | d<| jd| j| jdt| }|||d<qdS)zApply the decrypted signature to the stream manifest. :param dict stream_manifest: Details of the media streams available. :param str js: The contents of the base.js asset file. )rVr,r$r%UNKNOWN signaturesz&sig=z&lsig=zsignature found, skip decipher)Zciphered_signaturez+finished descrambling signature for itag=%sZitagcSsi|]\}}||dqS)rr)r]kvrrr sz#apply_signature..sigZ ratebypassnz://?N)r enumeraterXr*rr@rAZ get_signaturer r r2itemskeysr/Z calculate_nschemenetlocpathr ) rkrlrVcipheristreamr,Z live_streamrn parsed_urlZ query_paramsZ initial_nZnew_nrrrapply_signaturesJ      "r) stream_datar"cCsd|kr dSg}d|kr*||dd|krD||d|D]R}d|krd|krt|d}|dd|d<|dd|d<|dd k|d <qHtd |S) a-Apply various in-place transforms to YouTube's media stream data. Creates a ``list`` of dictionaries by string splitting on commas, then taking each list item, parsing it as a query string, converting it to a ``dict`` and unquoting the value. :param dict stream_data: Dictionary containing query string encoded values. **Example**: >>> d = {'foo': 'bar=1&var=test,em=5&t=url%20encoded'} >>> apply_descrambler(d, 'foo') >>> print(d) {'foo': [{'bar': '1', 'var': 'test'}, {'em': '5', 't': 'url encoded'}]} r,NformatsZadaptiveFormatsZsignatureCipherrrotypeZFORMAT_STREAM_TYPE_OTFZis_otfzapplying descrambler)rxextendr r*r@rA)rrdataZ cipher_urlrrrapply_descramblers     rc CsHddg}|D]*}zt||WStk r4Yq Xq tddddS)zExtract the ytInitialData json from the watch_html page. This mostly contains metadata necessary for rendering the page on-load, such as video information, copyright notices, etc. @param watch_html: Html of the watch page @return: z'window\[['\"]ytInitialData['\"]]\s*=\s*zytInitialData\s*=\s* initial_dataZinitial_data_patternr:Nrrrrr9r;rrrrs rc CsHddg}|D]*}zt||WStk r4Yq Xq tddddS)aExtract the ytInitialPlayerResponse json from the watch_html page. This mostly contains metadata necessary for rendering the page on-load, such as video information, copyright notices, etc. @param watch_html: Html of the watch page @return: z1window\[['\"]ytInitialPlayerResponse['\"]]\s*=\s*rer)Zinitial_player_response_patternr:Nrrrrrr)s r))r"c Csvz0|dddddddddd}Wn ttfk rPtgYSXtd d |}d d |D}t|S) u<Get the informational metadata for the video. e.g.: [ { 'Song': '강남스타일(Gangnam Style)', 'Artist': 'PSY', 'Album': 'PSY SIX RULES Pt.1', 'Licensed to YouTube by': 'YG Entertainment Inc. [...]' } ] :rtype: YouTubeMetadata contentsZtwoColumnWatchNextResultsrbr-ZvideoSecondaryInfoRendererZmetadataRowContainerZmetadataRowContainerRendererZrowscSs d|kS)NmetadataRowRenderer)rx)xrrr;zmetadata..cSsg|] }|dqS)rr)r]rrrrr^Aszmetadata..)rX IndexErrorrfilter)rZ metadata_rowsrrrmetadata!s4 r)9__doc__logging urllib.parser0r= collectionsrrtypingrrrrrr r r r Z pytube.cipherr Zpytube.exceptionsrrrZpytube.helpersrZpytube.metadatarZ pytube.parserrr getLogger__name__r@strrr r!boolr#r+r.r4r8rPrTrNrZr[rYrWrgrrrr)rrrrrsD      $ ."9)