Language Identification Tools

A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).

Most of these tools require training on a big corpus (see List of resources by language for corpora per language), but many come with some prebuilt language models.

Free Software

• LibTextCat http://software.wise-guys.nl/libtextcat/ (BSD license)
  • http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat
  • http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes
  • http://olivo.net/software/lc4j/ – a java implementation
  • http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat
  • https://github.com/crodas/PHPTextCat/ – a php module for libtextcat
  • https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat
• Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier (Apache license)
• Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)
  • doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google

Proprietary

• Google Language Identification API
• Lingua-Systems lid http://www.lingua-systems.com/language-identifier/

See also

• Language Identification (State of the art)
• English Wikipedia on Language detection
• TextCat competitors – list compiled by Gertjan van Noord