Sciweavers

CIMCA
2006
IEEE

Identification of Document Language is Not yet a Completely Solved Problem

13 years 5 months ago
Identification of Document Language is Not yet a Completely Solved Problem
Existing Language Identification (LID) approaches do reach 100% precision, in most common situations, when dealing with documents written in just one language, and when those documents are large enough. However, LID approaches do not provide a reliable solution for some situations: when there is need to discriminate the correct variant of the language used in a text, for example, European or Brazilian variants of Portuguese, UK or USA English variants, or any other language variants. Another hard context occur with small touristic advertisements on the web, addressing foreigners but using local language to name most local entities. In this paper, we present a fully statisticsbased LID approach which learns the most discriminant information according to each context, and identifies the correct language or language variant a text is written in. This methodology is shown to be correct for normal texts and maintains its robustness in hard LID contexts.
Joaquim Ferreira da Silva, Gabriel Pereira Lopes
Added 13 Oct 2010
Updated 13 Oct 2010
Type Conference
Year 2006
Where CIMCA
Authors Joaquim Ferreira da Silva, Gabriel Pereira Lopes
Comments (0)