Open Language Data Initiative

Contribution Guidelines

Thank you for your interest in contributing to the OLDI datasets.

Wheter you plan on contributing fixes to existing data or completely new translations, please open an issue in the relevant dataset repository, to ensure nobody else is already working on the same task. This will also allow the community to better coordinate work.

Typically, contributing a new translation involves starting from the original English data and having it translated by qualified, native speakers of the target language. We also ask contributors to fill out a data card containing details about the language being targeted, as well as information on how the translation was carried out. Please see the next section for more information.

Language data release checklist

Here are the steps which should be followed for all new contributions:

Language codes

We make use of standardized language codes such as eng_Latn. These codes are used throughout our projects to identify languages, as well as in filenames to indicate the language of their contents:

Data card

For new data, we collect precise information about the targeted language and dialect, as well as the translation workflow that was followed. Please use the following Markdown template to provide this information.


# Data card ## Data description <!-- Concisely describe the data associated with this card, for example: --> FLORES+ dev in Luxemburgish ## Data license <!-- New data contributions must be released under the same license as the parent dataset. Please specify the license using an SPDX license identifier. --> CC-BY-SA-4.0 ## Data attribution <!-- Who should be credited for creating this dataset? Feel free to include citation data in BibTeX format. --> ```bibtex @article{myarticle,   title={Something},   author={Somebody},   year={2024}, } ``` ## Language name <!-- What is the language endonym, i.e. the language name in the language itself? You may provide several names, e.g.: --> Lëtzebuergesch ## Language name in English <!-- What is the language name in English? You may provide several names, e.g.: --> Luxembourgish, Luxemburgish, Luxembourgian ## Language codes <!-- If this language is assigned an ISO 639-3 individual language code (not a macrolanguage code), specify it here. If this language is assigned a Glottocode, please specify it here. For example: --> * ISO 639-3: ltz * Glottocode: luxe1243 ## Script <!-- What script is this language written in? Please specify it using an ISO 15924 code, for example: --> * ISO 15924: Latn ## Dialectal and orthographic information <!-- Please provide information about dialectal and orthographic variation. Is this language likely to differ significantly between the communities or regions in which it is spoken/written? If applicable, specify which dialect(s) and orthography your contribution targets. --> ## Additional language information <!-- Any relevant additional information on the language, such as: a list of reference publications and software (dictionaries, grammars, spellcheckers) which target the language – and, if applicable, dialect(s) – of your contribution. --> ## Translation workflow <!-- What workflow was followed in translating this dataset? Relevant information includes: what language the content was translated from, the number of translators, aggregate translator information (how many were native speakers in the target language, how many were highly proficient in the target languages, how many had professional translation experience), was any fraction of the data checked independently by third parties, etc. Example: --> Data was translated from English by 5 translators, all native speakers of the target language and highly proficient in English (at C2 level of the European Language Framework). All translators were either professional translators or had relevant qualifications (university degrees in Translation and Interpreting or Linguistics). 100% of the data was checked by one more independent translator. ## Additional guidelines <!-- Were any additional guidelines provided to or agreed upon by translators? Examples might include style guidelines, the use of particular grammatical forms or sentence structures, specific spelling or punctuation rules to be followed, etc. -->

Translation guidelines

These translation guidelines must be acknlowedge by all translators who will be contributing data.

Important note

Your translations will be used to help train or evaluate Machine Translation engines. For this reason, this project requires Human Translation.

General guidelines

  1. You will be translating sentences coming from different sources. In some cases, a link to the source document might be provided to give you more context. If available, please refer to it.
  2. Do not convert any units of measurement. Translate them exactly as noted in the source content.
  3. When translating, please maintain the same tone used in the source document. For example, encyclopedic content coming from sources like Wikipedia should be translated using a formal tone.
  4. Provide fluent translations without deviating too much from the source structure. Only allow necessary changes.
  5. Do not expand or replace information compared to what is present in the source documents. Do not add any explanatory or parenthetical information, definitions, etc.
  6. Do not ignore any meaningful text that was present in the source.
  7. In case of multiple possible translations, please pick the one that makes the most sense (e.g., for gender concordance, cultural fit in the target language, level of formality, etc.).
  8. Translations must be faithful to the source in terms of pragmatics such as (if applicable) level of hedging/modality, sentiment and its intensity, negation, speech effects (disfluencies), etc.
  9. For proper nouns and common abbreviations, please see the guidelines on Named Entities below.
  10. Idiomatic expressions should not be translated word for word. Use an equivalent idiom, if one exists. If no equivalent idiom exists, use an idiom of similar meaning. If no similar expressions exist in the target language, paraphrase the idiom such that the meaning is retained in the target language.
  11. When a pronoun to be translated is ambiguous (for instance, when it could be interpreted as either him/her or he/she), opt for gender neutral pronouns (such as them/they) if those exist in the target language. However, when a pronoun to be translated is clearly marked for gender, you should follow the source material and continue to mark for gender.

Named entities

Named Entities are people, places, organisations, etc., that are commonly referred to using a proper noun. This section provides guidance on how to handle Named Entities. Please review the following guidelines carefully:

  1. If there is a commonly used term in the target language for the Named Entity:
    1. If the most commonly used term is the same as in the source language, then keep it as it is.
    2. If the most commonly used term is a translation or a transliteration, then use that.
  2. If there is no commonly used term:
    1. If possible, a transliteration of the original term should be used.
    2. If a transliteration would not be commonly understood in the context, and the source term would be more acceptable, you may retain the original term.