Linux and UNIX Man Pages

Linux & Unix Commands - Search Man Pages

unidesc(1) [debian man page]

unidesc(1)						      General Commands Manual							unidesc(1)

NAME
unidesc - Describe the contents of a Unicode text file SYNOPSIS
unidesc ([option flags]) (<file name>) If no input file name is supplied, unidesc reads from the standard input. DESCRIPTION
unidesc describes the content of a Unicode text file by reporting the character ranges to which different portions of the text belong. The ranges reported include both official Unicode ranges and the constructed language ranges within the Private Use Areas registered with the Conscript Unicode Registry (http://www.evertype.com/standards/csur/). For each range of characters, unidesc prints the character or byte offset of the beginning of the range, the character or byte offset of the end of the range, and the name of the range. Offsets start from 0. Since the ASCII digits, punctuation, and whitespace characters are frequently used by other writing systems, by default these characters are treated as neutral, that is, as not belonging exclusively to any particular character range. These characters are treated as belonging to the range of whatever characters precede them. If the input begins with neutral characters, they are treated as belonging to the range of whatever characters follow them. If the file consists entirely of neutral characters, the range is identified as Neutral followed by Basic Latin in square brackets. A magic number identifying the Unicode encoding is not part of the Unicode standard, so pure Unicode files do not contain a magic number. However, informal conventions have arisen for this purpose. If the command line flag -m is given, unidesc will attempt to identify the Unicode subtype by examining the first few bytes of the input. If the input is identified as one of the two acceptable types, UTF-8 or native order UTF-32, it will then proceed to describe the contents of the input. Otherwise, it will report what it has learned and exit. Note that if the file does contain a magic number, you must use the -m flag. Without this flag unidesc assumes that the input consists of pure Unicode with the character data beginning immediately. It will therefore be thrown off by the magic number. By default, input is expected to be UTF-8. Native order UTF-32 is also acceptable. UTF-32 may be specified via the command line flag -u or, if the command line flag -m is given, via the magic number. COMMAND LINE FLAGS
-b Give file offsets in bytes rather than characters. -d Treat the ASCII digits as belonging exclusively to the Basic Latin range. -h Print usage information. -L List the Unicode ranges alphabetically. -l List the Unicode ranges by codepoint. -m Check the file's magic number to determine the Unicode subtype. -p Treat ASCII punctuation as belonging exclusively to the Basic Latin range. -r Instead of listing ranges as they are encountered, just list the ranges detected after all input has been read. -u Input is native order UTF-32. -v Print version information. -w Treat ASCII whitespace as belonging exclusively to the Basic Latin range. SEE ALSO
uniname REFERENCES
Unicode Standard, version 5.0 AUTHOR
Bill Poser billposer@alum.mit.edu LICENSE
GNU General Public License June, 2007 unidesc(1)

Check Out this Related Man Page

unifuzz(1)						      General Commands Manual							unifuzz(1)

NAME
unifuzz - Emit strings designed to test Unicode handling SYNOPSIS
unifuzz ([option flags]) DESCRIPTION
unifuzz emits strings designed to test the ability of programs intended to accept Unicode input to handle unexpected input. These include: characters from all Unicode ranges, Private Use characters, surrogates, undefined characters, non-characters, control characters, exotic space characters, sequences violating normalization rules, unexpected sequences (e.g. a base character from one range followed by a combin- ing character from another range), and long sequences of combining characters. It can also generate very long lines, strings containing embedded nulls, and ill-formed UTF-8. COMMAND LINE FLAGS
-b Restrict the output to the Basic Multilingual Plane (Plane 0). -g Do not emit specific characters. -h Print usage information. -l Emit very long lines. -n Emit string with embedded nulls. -q Be quiet. Omit commentary. -r <number> Set the number of random characters to emit. -S Scan ranges - emit a character from each range. -s <seed> Set the seed for the random number generator. -u Emit ill-formed UTF-8. -v Print version information. The sequence of random characters is determined by a pseudorandom number generator, so the same sequence can be obtained by setting the seed to the same value. If not set on the command line, a seed is chosen based on the time of execution. The seed used is included in the output in a line of the form "Seed = NNNNNN" immediately preceding the random character sequence. Note that in order to obtain the same sequence it is necessary to keep the same setting for restriction of output to the BMP. REFERENCES
Unicode Standard, version 5.0 AUTHOR
Bill Poser billposer@alum.mit.edu LICENSE
GNU General Public License April, 2008 unifuzz(1)
Man Page