org.basex.util.ft

Class JapaneseTokenizer

java.lang.Object

org.basex.util.ft.FTIterator

org.basex.util.ft.Tokenizer

org.basex.util.ft.JapaneseTokenizer

All Implemented Interfaces:

java.lang.Comparable<org.basex.util.ft.LanguageImpl>, java.util.Iterator<FTSpan>

public class JapaneseTokenizer
extends Tokenizer

Japanese lexer using igo (http://igo.sourceforge.jp/).

Author:

BaseX Team 2005-12, BSD License, Toshio HIRAI

Field Summary

Fields inherited from class org.basex.util.ft.Tokenizer

IMPL

Constructor Summary

Constructors

Constructor and Description
JapaneseTokenizer(FTOpt fto) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
int	compareTo(org.basex.util.ft.LanguageImpl o)
boolean	equals(java.lang.Object o)
int	hashCode()
boolean	hasNext()
JapaneseTokenizer	init(byte[] txt) Initializes the iterator.
FTSpan	next()
byte[]	nextToken() Returns the next token.
protected byte	prec() Returns the precedence of the processor.
boolean	supports(Language ln) Checks if the specified language is supported.

Methods inherited from class org.basex.util.ft.Tokenizer

supportFor

Methods inherited from class org.basex.util.ft.FTIterator

remove

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JapaneseTokenizer

public JapaneseTokenizer(FTOpt fto)

Constructor.

Parameters:

fto - (optional) full-text options

Method Detail

init

public JapaneseTokenizer init(byte[] txt)

Description copied from class: FTIterator

Initializes the iterator.

Specified by:

init in class FTIterator

Parameters:

txt - text

Returns:

self reference

hasNext

public boolean hasNext()

next

public FTSpan next()

nextToken

public byte[] nextToken()

Description copied from class: FTIterator

Returns the next token. May be called as an alternative to Iterator.next() to avoid the creation of new FTSpan instances.

Specified by:

nextToken in class FTIterator

Returns:

token

prec

protected byte prec()

Returns the precedence of the processor. If two language implementations support the same language, the processor with the higher precedence will be selected.

Returns:

precedence

supports

public boolean supports(Language ln)

Checks if the specified language is supported.

Parameters:

ln - language

Returns:

true if language is supported

equals

public final boolean equals(java.lang.Object o)

Overrides:

equals in class java.lang.Object

compareTo

public final int compareTo(org.basex.util.ft.LanguageImpl o)

Specified by:

compareTo in interface java.lang.Comparable<org.basex.util.ft.LanguageImpl>

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object