Class HTML5::HTMLTokenizer
In: lib/html5/tokenizer.rb
Parent: Object

This class takes care of tokenizing HTML.

  • @current_token Holds the token that is currently being processed.
  • @state Holds a reference to the method to be invoked… XXX
  • @states Holds a mapping between states and methods that implement the state.
  • @stream Points to HTMLInputStream object.

Methods

after_attribute_name_state   after_doctype_name_state   after_doctype_public_identifier_state   after_doctype_system_identifier_state   attribute_name_state   attribute_value_double_quoted_state   attribute_value_single_quoted_state   attribute_value_unquoted_state   before_attribute_name_state   before_attribute_value_state   before_doctype_name_state   before_doctype_public_identifier_state   before_doctype_system_identifier_state   bogus_comment_state   bogus_doctype_state   close_tag_open_state   comment_end_dash_state   comment_end_state   comment_start_dash_state   comment_start_state   comment_state   consume_entity   consume_number_entity   data_state   doctype_name_state   doctype_public_identifier_double_quoted_state   doctype_public_identifier_single_quoted_state   doctype_state   doctype_system_identifier_double_quoted_state   doctype_system_identifier_single_quoted_state   each   emit_current_token   entity_data_state   markup_declaration_open_state   new   process_entity_in_attribute   process_solidus_in_tag   tag_name_state   tag_open_state  

Attributes

content_model_flag  [RW] 
current_token  [RW] 
stream  [R] 

Public Class methods

new(stream, options = {})

XXX need to fix documentation

Public Instance methods

consume_number_entity(isHex)

This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present @token_queue << {:type => :ParseError}" is invoked.

data_state()

XXX AT Perhaps we should have Hixie run some evaluation on billions of documents to figure out what the order of the various if and elsif statements should be.

each() {|:type => :ParseError, :data => @stream.errors.shift until @stream.errors.empty?| ...}

This is where the magic happens.

We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested.

emit_current_token()

This method is a generic handler for emitting the tags. It also sets the state to "data" because that‘s what‘s needed after a token has been emitted.

process_entity_in_attribute()

This method replaces the need for "entityInAttributeValueState".

process_solidus_in_tag()

If the next character is a ’>’, convert the current_token into an EmptyTag

[Validate]