Package ai.djl.basicdataset.nlp

Class TatoebaEnglishFrenchDataset

java.lang.Object

ai.djl.training.dataset.RandomAccessDataset

ai.djl.basicdataset.nlp.TextDataset

ai.djl.basicdataset.nlp.TatoebaEnglishFrenchDataset

All Implemented Interfaces:

ai.djl.training.dataset.Dataset

public class TatoebaEnglishFrenchDataset
extends TextDataset

TatoebaEnglishFrenchDataset is a English-French machine translation dataset from The Tatoeba Project (http://www.manythings.org/anki/).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	TatoebaEnglishFrenchDataset.Builder	A builder for a TatoebaEnglishFrenchDataset.

Nested classes/interfaces inherited from class ai.djl.basicdataset.nlp.TextDataset

TextDataset.Sample

Nested classes/interfaces inherited from class ai.djl.training.dataset.RandomAccessDataset

ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<T extends ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<T>>

Nested classes/interfaces inherited from interface ai.djl.training.dataset.Dataset

ai.djl.training.dataset.Dataset.Usage

Field Summary

Fields inherited from class ai.djl.basicdataset.nlp.TextDataset

manager, mrl, prepared, samples, sourceTextData, targetTextData, usage

Fields inherited from class ai.djl.training.dataset.RandomAccessDataset

dataBatchifier, device, labelBatchifier, limit, pipeline, prefetchNumber, sampler, targetPipeline

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	TatoebaEnglishFrenchDataset(TatoebaEnglishFrenchDataset.Builder builder)	Creates a new instance of TatoebaEnglishFrenchDataset.

Method Summary

Modifier and Type	Method	Description
protected long	availableSize()
static TatoebaEnglishFrenchDataset.Builder	builder()	Creates a new builder to build a TatoebaEnglishFrenchDataset.
ai.djl.training.dataset.Record	get(ai.djl.ndarray.NDManager manager, long index)
void	prepare(ai.djl.util.Progress progress)

Methods inherited from class ai.djl.basicdataset.nlp.TextDataset

getProcessedText, getRawText, getSamples, getTextEmbedding, getVocabulary, preprocess

Methods inherited from class ai.djl.training.dataset.RandomAccessDataset

getData, getData, getData, getData, newSubDataset, newSubDataset, randomSplit, size, subDataset, subDataset, subDataset, subDataset, toArray

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface ai.djl.training.dataset.Dataset

matchingTranslatorOptions, prepare

Constructor Details

TatoebaEnglishFrenchDataset

protected TatoebaEnglishFrenchDataset(TatoebaEnglishFrenchDataset.Builder builder)

Creates a new instance of TatoebaEnglishFrenchDataset.

Parameters:

builder - the builder object to build from

Method Details

builder

public static TatoebaEnglishFrenchDataset.Builder builder()

Creates a new builder to build a TatoebaEnglishFrenchDataset.

Returns:

a new builder

prepare

public void prepare(ai.djl.util.Progress progress)
             throws IOException,
ai.djl.modality.nlp.embedding.EmbeddingException

Throws:

IOException

ai.djl.modality.nlp.embedding.EmbeddingException

get

public ai.djl.training.dataset.Record get(ai.djl.ndarray.NDManager manager,
 long index)

Specified by:

get in class ai.djl.training.dataset.RandomAccessDataset

availableSize

protected long availableSize()

Specified by:

availableSize in class ai.djl.training.dataset.RandomAccessDataset