lines_to_tokenlists() -> new lines_to_tokenlists component.
Takes individual lines of text and separates them into white space separated tokens. Tokens can be enclosed with single or double quote marks.
Warning! You should be using the inbox/outbox interface, not these methods (except construction). This documentation is designed as a roadmap as to their functionalilty for maintainers and new component developers. |