mmseg(1) [debian man page]

MMSEG(1)						User Contributed Perl Documentation						  MMSEG(1)

NAME

       mmseg - maximum matching segment Chinese text.

SYNOPSIS

       mmseg -d dict_file [option]... [corpus_file]...

DESCRIPTION

       mmseg is a tool for segmenting Chinese text into words using maximum matching algorithm. mmseg segments corpus_file, or standard input if
       no filename is specified, and write the segmented result to standard output.

OPTIONS

       -d dict_file
	   Use dict_file as lexicon. A default lexicon can be found at /usr/share/sunpinyin-slm/dict.utf8.

       -f,--format (text|bin)
	   Output Format, can be 'text' or 'bin'. default 'bin'.  Normally, in text mode, word text are output, while in binary mode, binary short
	   integer of the word-ids are written to stdout.

       -s, --stok STOK_ID
	   Sentence token id. Default 10.  It will be written to output in binary mode after every sentence.

       -i, --show-id
	   Show Id info. Under text output format mode, attach id after known words.  If under binary mode, print id(s) in text.

       -a, --ambiguious-id AMBI-ID
	   Ambiguious means ABC => A BC or AB C. If specified (AMBI-ID != 0), The sequence ABC will not be segmented, in binary mode, the AMBI-ID
	   is written out; in text mode, "<ambi>ABC</ambi>" will be output. Default is 0.

NOTES

       Under binary mode, consecutive id of 0 are merged into one 0.  Under text mode, no space are inserted between unknown-words.

AUTHOR

       Originally written by Phill.Zhang <phill.zhang@sun.com>.  Currently maintained by Kov.Chai <tchaikov@gmail.com>.

SEE ALSO

       slmseg(1), ids2ngram (1).

perl v5.14.2							    2012-06-09								  MMSEG(1)

SLMBUILD(1)						User Contributed Perl Documentation					       SLMBUILD(1)

NAME

       slmbuild - generate language model from idngram file

SYNOPSIS

       slmbuild [option]... idngram_file...

DESCRIPTION

       slmbuild generates a back-off smoothing language model from a given idngram file. Generally, the idngram_file is created by ids2ngram.

OPTIONS All the following options are mandatory.
       -n,--NMax N
	   1 for unigram, 2 for bigram, 3 for trigram. Any number not in the range of 1..3 is not valid.

       -o, --out output-file
	   Specify the output xfilei name.

       -l, --log
	   using -log(pr), use pr directly by default.

       -w, --wordcount N
	   Lexican size, number of different words.

       -b, --brk id...
	   Set the ids which should be treated as breaker.

       -e, --e id...
	   Set the ids which should not be put into LM.

       -c, --cut c...
	   k-grams whose freq <= c[k] are dropped.

       -d, --discount method, param...
	   The k-th -d parm specifies the discount method

	   For k-gram, possibble values for method/param are:

		 B<GT>,I<R>,I<dis>  : B<GT> discount for r E<lt>= I<R>, r is the freq of a ngram.
			     Linear discount for those r E<gt> I<R>, i.e. r'=r*dis
			     0 E<lt>E<lt> dis E<lt> 1.0, for example 0.999
		 B<ABS>,[I<dis>] : Absolute discount r'=r-I<dis>. And I<dis> is optional
			     0 E<lt>E<lt> I<dis> E<lt> cut[k]+1.0, normally I<dis> E<lt> 1.0.
		 LIN,[I<dis>] : Linear discount r'=r*dis. And dis is optional
			     0 E<lt> dis E<lt> 1.0

NOTE

       -n must be given before -c -b. And -c must give right number of cut-off, also -ds must appear exactly N times specifying the discounts for
       1-gram, 2-gram..., respectively.

       BREAKER-IDs could be SentenceTokens or ParagraphTokens. Conceptually, these ids have no meaning when they appeared in the middle of n-gram.

       EXCLUDE-IDs could be ambiguious-ids. Conceptually, n-grams which contain those ids are meaningless.

       We can not erase ngrams according to BREAKER-IDS and EXCLUDE-IDs directly from IDNGRAM file, because some low-level information is still
       useful in it.

EXAMPLE

       Following example read 'all.id3gram' and write trigram model 'all.slm'.

       At 1-gram level, use Good-Turing discount with cut-off 0, i<R>=8, dis=0.9995. At 2-gram level, use Absolute discount with cut-off 3, dis
       auto-calc. At 3-gram level, use Absolute discount with cut-off 2, dis auto-calc. Word id 10,11,12 are breakers (sentence/para/paper
       breaker, etc). Exclude-ID is 9. Lexicon contains 200000 words. The result languagme model uses -log(pr).

       slmbuild -l -n 3 -o all.slm -w 200000 -c 0,3,2 -d GT,8,0.9995 -d ABS -d ABS -b 10,11,12 -e 9 all.id3gram

AUTHOR

       Originally written by Phill.Zhang <phill.zhang@sun.com>.  Currently maintained by Kov.Chai <tchaikov@gmail.com>.

SEE ALSO

       ids2ngram(1), slmprune(1).

perl v5.14.2							    2012-06-09							       SLMBUILD(1)