U
}gE @ s" d Z ddlmZ dZdgZddlmZ ddlmZm Z m
Z
m Z m Z m
Z
mZmZmZmZmZ ddlmZmZmZmZmZmZ ddlmZm Z dd l m Z m Z m Z m!Z! dd
l"m#Z# erdd l$m%Z% dd lm&Z& dd
l'm(Z(m)Z)m*Z* dZ+e e e,e,f e,e,gdf Z-G dd dee Z.G dd de Z/dS )zCUse the HTMLParser library to parse HTML files that aren't too bad. ) annotationsMITHTMLParserTreeBuilder)
HTMLParser) AnyCallablecastDictIterableListOptional
TYPE_CHECKINGTupleTypeUnion)
AttributeDictCDataComment DeclarationDoctypeProcessingInstruction)EntitySubstitution
UnicodeDammit)DetectsXMLParsedAsHTMLHTMLHTMLTreeBuilderSTRICTParserRejectedMarkup)
BeautifulSoup)NavigableString) _Encoding
_Encodings
_RawMarkupz html.parserNc @ s e Zd ZU dZded< dZded< edddd dd
d d Zd ed
< ded< ded< dddddZddddddZd2dddddd d Z d3dddd d d Z
ddd!d"d#Z ddd$d%d&Z ddd$d'd(Z
ddd!d)d*Zddd!d+d,Zddd!d-d.Zddd!d/d0Zd1S )4BeautifulSoupHTMLParserreplacestrREPLACEignoreIGNOREon_duplicate_attributer r z&Union[str, _DuplicateAttributeHandler])soupargsr+ kwargsc O s: || _ || _|jj| _tj| f|| g | _| d S N)r, r+ builderattribute_dict_classr __init__ already_closed_empty_elementZ_initialize_xml_detector)selfr, r+ r- r. r5 ;/tmp/pip-unpacked-wheel-kgiupv3k/bs4/builder/_htmlparser.pyr2 T s
z BeautifulSoupHTMLParser.__init__r+ z List[str]r3 r, None)messagereturnc C s t |d S r/ r )r4 r8 r5 r5 r6 erroro s z BeautifulSoupHTMLParser.errorz List[Tuple[str, Optional[str]]])nameattrsr9 c C s | j ||dd | | dS )zHandle an incoming empty-element tag.
html.parser only calls this method when the markup looks like
.
F)handle_empty_elementN)handle_starttag
handle_endtag)r4 r; r< r5 r5 r6 handle_startendtag s z*BeautifulSoupHTMLParser.handle_startendtagTbool)r; r< r= r9 c C s | }|D ]f\}}|dkr d}||krj| j}|| jkr:qr|d| jfkrR|||< qrtt|}|||| q |||< q | jjjr| \}} nd }} | jj
|dd||| d}
|
r|
j r|r| j |dd | j
| | jdkr| | dS )zHandle an opening tag, e.g. ''
:param handle_empty_element: True if this tag is known to be
an empty-element tag (i.e. there is not expected to be any
closing tag).
N )
sourceline sourceposF)check_already_closed)r1 r+ r) r' r _DuplicateAttributeHandlerr, r0 Zstore_line_numbersgetposr> Zis_empty_elementr? r3 appendZ_root_tag_nameZ_root_tag_encountered) r4 r; r< r= Z attr_dictkeyvalueZon_duperC rD tagr5 r5 r6 r> s:
z'BeautifulSoupHTMLParser.handle_starttag)r; rE r9 c C s, |r || j kr | j | n | j| dS )zHandle a closing tag, e.g. ''
:param name: A tag name.
:param check_already_closed: True if this tag is expected to
be the closing portion of an empty-element tag,
e.g. ''.
N)r3 remover, r? )r4 r; rE r5 r5 r6 r? s z%BeautifulSoupHTMLParser.handle_endtag)datar9 c C s | j | dS )z4Handle some textual data that shows up between tags.N)r, handle_datar4 rM r5 r5 r6 rN s z#BeautifulSoupHTMLParser.handle_data)r; r9 c C s | dr t|dd}n$| dr8t|dd}nt|}d}|dk r| jjdfD ]4}|sbqXzt|g|}W qX tk
r Y qXX qX|sz t|}W n t t
fk
r Y nX |pd}| | dS )zHandle a numeric character reference by converting it to the
corresponding Unicode character and treating it as textual
data.
:param name: Character number, possibly in hexadecimal.
x XN z windows-1252u �)
startswithintlstripr, original_encoding bytearraydecodeUnicodeDecodeErrorchr
ValueError
OverflowErrorrN )r4 r; Z real_namerM encodingr5 r5 r6 handle_charref s*
z&BeautifulSoupHTMLParser.handle_charrefc C s0 t j|}|dk r|}nd| }| | dS )zHandle a named entity reference by converting it to the
corresponding Unicode character(s) and treating it as textual
data.
:param name: Name of the entity reference.
Nz&%s)r ZHTML_ENTITY_TO_CHARACTERgetrN )r4 r; characterrM r5 r5 r6 handle_entityref
s
z(BeautifulSoupHTMLParser.handle_entityrefc C s& | j | j | | j t dS )zOHandle an HTML comment.
:param data: The text of the comment.
N)r, endDatarN r rO r5 r5 r6 handle_comment s
z&BeautifulSoupHTMLParser.handle_commentc C s6 | j |tdd }| j | | j t dS )zYHandle a DOCTYPE declaration.
:param data: The text of the declaration.
zDOCTYPE N)r, rc lenrN r rO r5 r5 r6 handle_decl& s
z#BeautifulSoupHTMLParser.handle_declc C sN | dr$t}|tdd }nt}| j | j| | j| dS )z{Handle a declaration of unknown type -- probably a CDATA block.
:param data: The text of the declaration.
zCDATA[N)upperrT r re r r, rc rN )r4 rM clsr5 r5 r6 unknown_decl0 s
z$BeautifulSoupHTMLParser.unknown_declc C s0 | j | j | | | | j t dS )z\Handle a processing instruction.
:param data: The text of the instruction.
N)r, rc rN Z_document_might_be_xmlr rO r5 r5 r6 handle_pi? s
z!BeautifulSoupHTMLParser.handle_piN)T)T)__name__
__module__ __qualname__r' __annotations__r) r2 r: r@ r> r? rN r_ rb rd rf ri rj r5 r5 r5 r6 r$ = s&
>(
r$ c s e Zd ZU dZdZded< dZded< eZded< ee e
gZ d ed
< d ed < dZ ded
< d dddd fdd
Z
d!ddddddddZdd d d d Z ZS )"r zA Beautiful soup `bs4.builder.TreeBuilder` that uses the
:py:class:`html.parser.HTMLParser` parser, found in the Python
standard library.
FrA is_xmlT picklabler& NAMEz
Iterable[str]featuresz$Tuple[Iterable[Any], Dict[str, Any]] parser_argsTRACKS_LINE_NUMBERSNzOptional[Iterable[Any]]zOptional[Dict[str, Any]]r )rs
parser_kwargsr. c sl t }dD ] }||kr
||}|||< q
tt| jf | |pBg }|pJi }|| d|d< ||f| _dS )a Constructor.
:param parser_args: Positional arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param parser_kwargs: Keyword arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param kwargs: Keyword arguments for the superclass constructor.
r* Fconvert_charrefsN)dictpopsuperr r2 updaters )r4 rs ru r. Zextra_parser_kwargsargrJ __class__r5 r6 r2 [ s
z HTMLParserTreeBuilder.__init__r# zOptional[_Encoding]zOptional[_Encodings]zDIterable[Tuple[str, Optional[_Encoding], Optional[_Encoding], bool]])markupuser_specified_encodingdocument_declared_encodingexclude_encodingsr9 c c s t |tr |dddfV dS g }|r.|| g }|r@|| t|||d|d}|jdkrftdn|j|j|j|jfV dS )a2 Run any preliminary steps necessary to make incoming markup
acceptable to the parser.
:param markup: Some markup -- probably a bytestring.
:param user_specified_encoding: The user asked to try this encoding.
:param document_declared_encoding: The markup itself claims to be
in this encoding.
:param exclude_encodings: The user asked _not_ to try any of
these encodings.
:yield: A series of 4-tuples: (markup, encoding, declared encoding,
has undergone character replacement)
Each 4-tuple represents a strategy for parsing the document.
This TreeBuilder uses Unicode, Dammit to convert the markup
into Unicode, so the ``markup`` element of the tuple will
always be a string.
NFT)known_definite_encodingsuser_encodingsZis_htmlr zPCould not convert input to Unicode, and html.parser will not accept bytestrings.)
isinstancer& rH r Zunicode_markupr rW Zdeclared_html_encodingZ contains_replacement_characters)r4 r~ r r r r r Zdammitr5 r5 r6 prepare_markupy s2
z$HTMLParserTreeBuilder.prepare_markupr7 )r~ r9 c
C s | j \}}t|tst| jd k s&tt| jf||}z|| | W n* tk
rx } z t|W 5 d }~X Y nX g |_ d S r/ )
rs r r& AssertionErrorr, r$ feedcloser r3 )r4 r~ r- r. parserer5 r5 r6 r s
zHTMLParserTreeBuilder.feed)NN)NNN)rk rl rm __doc__ro rn rp
HTMLPARSERrq r r rr rt r2 r r
__classcell__r5 r5 r| r6 r J s
! H)0r
__future__r __license____all__ html.parserr typingr r r r r
r r r
r r r Z bs4.elementr r r r r r Z
bs4.dammitr r Z bs4.builderr r r r Zbs4.exceptionsr Zbs4r r Z bs4._typingr! r" r# r r& rF r$ r r5 r5 r5 r6 s( 4