GSMA and Pleias Launch Open Source AI Model Identifying 50 African Languages

The GSMA — global association of mobile operators — in partnership with the French startup Pleas launched a language identification model (LDI) based on artificial intelligence, totally open-source. The system is capable of recognizing more than 200 languages, especially 50 African languages that are historically ignored by large language models. The initiative, recently announced, represents a concrete step towards reducing linguistic bias in AI technologies and expanding access to digital services in regions where languages such as Swahili, Yoruba, Hausa and Zulu are major.

According to the original source of Developing Telecoms, the model was trained with a diverse set of data, ensuring accuracy even in languages with few digital resources.

Open-source AI model: How do you identify 50 African languages?

The LID uses deep learning techniques to classify text or audio fragments into more than 200 language categories. Unlike generic models that prioritize English, Chinese and some European languages, the Pleias model was trained with specific corpora of under-represented African languages. The architecture is lightweight and efficient, allowing execution on devices with limited resources — ideal for the African mobile scenario, where most of the population accesses the internet exclusively by mobile phone. Source code, trained weights and documentation are available in open repositories, encouraging collaboration of the developer community.

Why is the inclusion of African languages crucial for the future of AI?

Large language models, such as GPT-4 and Claude, have significantly lower performance in non-Western languages, perpetuating a digital exclusion cycle. It is estimated that more than 2,000 languages are spoken in Africa, but less than 1% of popular model training data contemplate these languages. Without robust identification and translation tools, millions of people are on the verge of innovations in health, education and finance. The Pleias and GSMA model seeks to reverse this framework by providing a basis for virtual assistants, customer service systems and automatic translators adapted to the linguistic diversity of the continent.

What does this open-source AI model mean for Brazilian developers and companies?

Although focused on Africa, the model can be adapted to other under-represented languages, including Brazilian indigenous languages or regional Portuguese variations. Technology companies in Brazil can use open source as a starting point to build more inclusive NLP solutions, integrating it into platforms such as [Amazon Quick: AWS launches desktop AI wizard who learns his work and creates live dashboards (/noticias/amazon-quick-aws-assistente-ia-desktop-aprende-trabalho-dashboards), who already seeks to democratize the use of AI in the corporate environment. The open-source approach accelerates local innovation without relying exclusively on foreign giants.

Frequently Asked Questions about the open source AI model identifying 50 African languages

How many languages can the model identify?

The model is capable of identifying more than 200 languages, including 50 African languages such as Swahili, Yoruba, Hausa, Zulu and Amharic.

The model is really open-source and where can I access it?

Yes, the model is fully open-source, with code, weights and documentation available in public repositories of Pleias and GSMA, accessible via GitHub and other AI platforms.

How does it compare to other language identification models?

The differential is in the coverage of under-represented African languages and in computational efficiency, being lighter than models such as fastText and more precise for languages with few available data.