LingDoc

LingDoc is a set of command line tools for Document and Language Engineering. They provide a head start for the development of solutions, be it for individual developers, service providers or system integrators. The source code is free. Separate GUI?s give access to the tools. LingDoc Transform is a general-purpose tool for the analysis of natural language texts, for XML up-conversion and for information extraction. It has been used in a wide variety of projects like the parsing of natural languages, transformation of graphemes into phonemes, classification of Latin words and conversion of a Russian-Dutch dictionary into a Dutch-Russian one. A more mundane task is the preprocessing of texts. Transform has two modes: one for pattern-matching and -replacement and one for parsing and transduction according to grammars and lexicons (lingware). Transform is based on unique extensions to the theory of formal grammars and automata. In essence, a document is treated like a sentence in a linguistic way. Conversion is treated like Machine Translation. Transform can be used for both processes. For XML up-conversion, the structure of the source document is specified by a formal grammar, akin to the document schema of the target document. In a first stage errors and inconsistencies in the source document are detected. In a second stage a correct target document is created. The grammar formalism supports regular expressions on the character and on the rule level, Chomsky type 0 rules, attributed variables, pattern rewriting rules and programming statements. Cascaded grammars may be used in a multi-stage/pipelined process, e.g. for the integration of linguistic layers and for the modularization of large transduction grammars. The generated programs have the unlimited look-ahead property and the on-line property, handle all kind of ambiguities and report errors as soon as possible. LingDoc Clarity helps authors to write clearly and consistently, according to a predefined standard, described by lingware. Clarity can also improve the quality of the output of commercial Machine Translation systems. If the language is sufficiently restricted, then Clarity can perform the translation by itself. Clarity consists of three main modules: Author, Capri and LexBench. Clarity Author assists an author during normal editing tasks with correction and standardization of spelling, terminology, grammar and style. If errors are encountered, suggestions for corrections are given. Author comes in three versions: LEditor, a standalone editor; FmEdit, an add-in for Framemaker; WoEdit, an add-in for Microsoft Word. Clarity Capri is an engine which performs the analysis, correction and translation; it runs in the background of Author. It can also run independently as an analyzer or a translator. The grammar formalism of Capri is rich enough for rules expressing readability, translatability, standardization and error reporting and correction. Clarity Lexbench is a workbench for the creation and maintenance of lexicons and thesauruses. It supports multiword?s and idiomatic expressions. It uses a thesaurus with preferred terms. Finally, the third tool LingDoc RevXml tracks all changes between two XML or SGML-tagged documents. The tool generates revision markers (indicating addition, deletion and change) in the revised document, which themselves take the form of additional XML tags. RevXml takes into account the tree structure of the documents, the attributes and the text in between tags. The basic unit of a text may be a character, a word, a sentence or the complete text in between two tags. No document schema is required. The GUI for Transform is available through RegNow, the other executables, the source code and documentation may be obtained for free from www.lingdoc.eu. Clarity and its GUI will be added in the near future when the User Manual is finished.