Machine Translation with Seq2Seq Model with

Attention Mechanism
Vigneshkumar M Dr. K.S.Kannan Jaswanth teja alli
Associate Professor Associate Professor 99220041097
Department of Computer Department of Computer Computer Science and
Science and Engineering, Science and Engineering, Engineering
Kalasalingam Academy of Kalasalingam Academy of Kalasalingam Academy of
Research and Education Research and Education Research and Education,
Krishnankoil, Virudhunagar Krishnankoil, Virudhunagar Krishnankoil
[email protected] [email protected] [email protected]

Doondi Surya vikram balla Teja nagendra prasad chakali Ram mohan reddy emani
99220041121 99220041148 99220041066
Computer Science and Computer Science and Computer Science and
Engineering Engineering Engineering
Kalasalingam Academy of Kalasalingam Academy of Kalasalingam Academy of
Research and Education, Research and Education, Research and Education,
Krishnankoil Krishnankoil Krishnankoil
[email protected] [email protected] [email protected]

Abstract: This is an implementation of an English-to- translation, pytorch, low-resource languages, dynamic

Hindi Seq2Seq model with Attention mechanism using context
PyTorch for deep learning. This model is based on the
encoder-decoder architecture in which the encoder I. INTRODUCTION
processes the input in English and the decoder produces
the Hindi translation. This Attention mechanism is used This paper, improving neural machine translation for low
to focus the attention on relevant parts of the source resource algerian dialect through transductive transfer
sentence during decoding so that it overcomes limitations learning strategy, attempts to look at transductive transfer
in long-sequence translation. This would include learning in order to improve the machine translation systems
tokenization, vocabulary building, and sequence of less available data of the algerian arabic dialect. The paper
conversion. For this kind of model training, the input text uses the madar corpus-a larger corpus of multi-dialectal
as well as the target text would then be transformed into arabic which transmits know-how-across to fine-tune on the
indexed sequences. small-scoped biased arabic dialect corpus of padic. The
accuracy showed some notable improvements with the
The encoder will have an embedding layer as well as an increase in the bleu results and indicated a possibility of
LSTM layer processing the input sentence, and the application in a low-resource context. To the best of the
hidden states produced whereas the decoder also has an author's knowledge, this work is the very first work on the
embedding layer as well as an LSTM layer and is using application of transductive transfer learning in Arabic
weights attention to capture context dynamically. The dialectal translation. [1]
model uses teacher forcing during the training, but it
decreases the probability as the epochs progress so that it This paper introduces a new self-adaptive mechanism in
is possible to learn the model's predictions step by step. neural machine translation referred to as self-adaptive
Training will run for some number of epochs, tracking control of temperature, the temperature being the amount of
Cross Entropy loss as a measure of model performance. softness applied to the attention based on context.
A translate function would draw inference by Conventional attention mechanisms typically make use of
constructing a sentence in Hindi from any string input in uniform distributions on attention that are inefficiently
English. All of the code is part of a complete strategy for applied to content and function words. SACT dynamically
the building of a neural machine translation model that controls the attention temperature, which sharpens the focus
is competent in low-resource settings when paying on content words and softens it on function words.
attention to the words most relevant to sentences' Experimental results on Chinese English and English
contexts, with huge improvement over standard models Vietnamese translations show that sact-enhanced models
without attention. outperform baseline seq2seq models, achieving BLEU score
improvements of 2.94 and 2.19, respectively. The
experiments prove that SACT helps use context more
Keyword's: seq2seq model, attention mechanism, effectively in order to improve translation quality. [2]
neural machine translation, encoder-decoder architecture,
english-hindi translation, teacher forcing, long-sequence The paper "Entity Highlight Generation as Statistical
and Neural Machine Translation" looks at automated
approaches of generating concise highlights of individual AVSR, which testifies to its efficacy against robust speech
entities from descriptive sentences, which may in turn find recognition tasks. [6]
several applications, including search engine result
enrichment. The paper proposes two approaches-a Statistical ALGORITHMS USED:
Machine Translation (SMT) model and a neural Sequence-
to-Sequence (Seq2Seq) model-the latter endowed with It uses two fundamental algorithms: the Seq2Seq model
attention, copy, and coverage mechanisms to further enhance and the Attention mechanism, which are basic to NMT. The
the accuracy of purported highlighted phrases by attended Seq2Seq model is an encoder-decoder architecture. It accepts
words, by the inclusion of relevant salient words, and in the an input language sequence, processes it into a hidden
prevention of redundancy. The inclusion of entity names in representation, and the decoder transforms it into a target
the decoder explains the fact that highlights created by the language sequence. In this scenario, the encoder employs a
model are more relevant. Experiments using automatically long short-term memory network. This network can capture
collected data indicate that Seq2Seq with the application of long dependencies in the sequence, so it maintains its context
attention and copy mechanisms most significantly for longer sentences. LSTMs are also used in the decoder,
outperforms SMT on a range of metrics, demonstrating the where the decoder translates the sequence word by word
merits of neural approaches for generating meaningful following the context of the encoder. Adding attention helps
descriptions of entities. [3]. the model select parts of the input sequence to focus on to
produce each word of the output while generating the output
In this paper, we propose an extension to sequence-to- in sequence. This is helpful when translating more or
sequence neural network architecture called the "Scratchpad syntactically more complex sentences since the model can
Mechanism", which allows the decoder to write interactively "attend" on input words of relevance on-the-fly. Teacher
onto the encoder layers, thus allowing it to act like a kind of Forcing during training ensures the same switches are
working memory for keeping track of produced content and applied to be enforced between the target word and the
further regulating its output. It is evaluated by a number of model's prediction, thereby gradually deposing on the actual
tasks including machine translation, question generation, and labels of its choice. These in parallel give the model room to
text summarization, achieving state-of-the-art or competitive shift focus dynamically. Its previous context builds better
results in entire datasets while improving the fluency and accuracy in the outputting words in translation. This hybrid
expressiveness in generated text.Results demonstrate the architecture is very efficient for neural machine translation
model's ability to perform well without large-scale tuning, tasks, especially for low-resource languages or very complex
thus establishing versatility for conditional language sentence structures in which the contextual relationship
generation scenarios. [4] between words must be understood.

The present paper, reflecting on Transparent Attention, II. LITERATURE SURVEY

provides an approach to improve deeper neural machine
translation model training. A modification of the attention The authors propose here a joint seq2seq model that can
mechanism is introduced, making error gradients propagate classify dialogue acts for conversational agents, showing a
faster across several layers against the common problems in new kind of guided attention mechanism together with a
training deep structures, such as vanishing gradients. The hierarchical encoder that couples textual and contextual
authors utilize this method for the Transformer and RNMT+ features from dialogue into one single modeling. By allowing
models, up to 20-layer deep training of encoders, resulting in for local and global tag dependencies without handcrafted
improvements in BLEU scores on standard benchmarks for features, the studied system outperformed other traditional
English-German and Czech-English translation tasks. methods in the DA classification benchmarks. The approach
Transparent Attention contains a promise for improving puts an extension on the previously advanced DA prediction
quality in translation when coupled with deeper models, technique in that it performs with an accuracy of 85% on the
hence reducing optimization difficulties that traditional Switchboard Dialogue Act Corpus and 91.6% on the MRDA.
depth presents. [5] [7]
The proposed paper introduces an entirely new approach Under the title "P-Transformer: Towards Better
to the study of audio-visual speech recognition that would Document-to-Document Neural Machine Translation," a
use the attention mechanism in a dual-cross-modality model position-aware Transformer model was developed which is
with transformers. The DCM attention scheme enhances intended to enhance the efficiency of document-to-document
recognition by leveraging both audio and visual inputs to act (Doc2Doc) neural machine translation (NMT). This novel
as queries, facilitating performance, especially in noisy model, named the P-Transformer, introduces both absolute
environments. In addition, the hybrid CTC/attention and relative position information in the attention mechanism,
architecture incorporated into the model enables monotonic mitigating the problems of position information loss that
alignment, which is paramount to accurately supporting hinder training on small datasets. The experimental results
speech recognition. The experiments were conducted using pronounce state-of-the-art results by P-Transformer in
LRS2-BBC and LRS3-TED datasets to show that the various language pairs and sizes of datasets involving
proposed method is able to yield word error rates lower than translations with utmost coherence and discourse-level
those achieved by existing transformer-based models for context capture. The approach shows impressive
improvements in BLEU scores and discourse phenomena III. EXISTING SYSTEM
against the baseline models. [8]
1. Google's Neural Machine Translation (GNMT)
This paper by Chengsong Lu, "Performance analysis of System
attention mechanism and teacher forcing ratio based on Overview: Perhaps the most commonly implemented
machine translation," analyzes results of various attention system, Google's Neural Machine Translation (GNMT)
mechanisms and teacher forcing ratios in models of machine
system has an original RNN-based Seq2Seq model with
translation. The comparison is made between Synthetic
attention. This enabled Google to replace their phrase-based
Attention and Multi-head Attention inside a sequence-to-
sequence framework. This research finds that Multi-head system with an end-to-end neural model that could generate
Attention results in higher accuracy at the price of increased fluent translations.
computational complexity. This leads to the conclusion that Architecture: GNMT was actually employing 8 layer
varied forcing ratios for the teacher imply that while a greater LSTM architecture for encoder and decoder. It was enabling
ratio accelerates convergence, it increases the risk of the model to focus on different parts of the input sentence
overfitting. In summary, the results of this paper illustrate that would improve its ability to translate even longer and
that Multi-head Attention outperforms Synthetic Attention more complex sentences with considerable margins.
and also shows an imbalance in the teacher forcing that needs Advantages: The Seq2Seq model of GNMT with attention
to be corrected to achieve stability in the model and its is producing much more natural and human-like translations
performance. [9] than other rule-based or phrase-based methods.
Pros: Despite such power to such an architecture, it was
This paper proposes a neural machine learning approach very vulnerable to very long sentences and required high
to workload prediction in cloud resources, specifically an computational resources for training.
Attention-based Seq2Seq model. Workload prediction is
OpenNMT
framed as a translation task with the attention mechanism to
improve prediction accuracy for time-series data.
Experimental results on real workloads gathered from a 2. OpenNMT
Google cluster show that the model achieves a 98.1% Overview: OpenNMT is an open-source neural machine
accuracy rate, modeling with CTMC and LSTM approaches. translation framework that is based on the Seq2Seq
In addition, the authors propose a very innovative cumulative architecture with attention. It was developed by the Harvard
validation strategy, which reduces computational time by NLP group and SYSTRAN to be efficient and highly
57%. [10] adaptable. Besides RNN-based Seq2Seq models with
attention, this enables it to apply others from the
This paper solves the molecular optimization problem Transformer family altogether.
by deep neural networks that could capture the chemist's Architecture: Initially supported by a bi-directional RNN
intuition by using matched molecular pairs (MMPs) and encoder with attention, it opened up flexible avenues for
reframing the molecular optimization task as a machine adaptation to various translation tasks and languages.
translation task. Two models, Seq2Seq with Attention and Advantages: are ease of deployment in production settings,
the Transformer, were trained to produce molecules with optimized for both research and commercial use. OpenNMT
optimized logD, solubility, and clearance by learning from
supports a wide array of customizations.
MMPs expressed in SMILES notation. The Transformer has
Disadvantages: Other RNN-based systems suffered in
better performance than other models in generating new
molecules with desirable properties and minimal structural terms of low efficiency on huge datasets with longer
changes, therefore assisting in the designer's intuitive process sentences until the advent of Transformer-based models.
for chemists. [11]
3. Microsoft's Neural Machine Translation System
This paper describes an approach based on Deep Overview: Before that, the earlier reliance of Microsoft had
Learning for the solution of Turkish math word problems - been upon the Seq2Seq models with attention for Bing
MWPs-performed by means of sequence-to-sequence -or Translator.
seq2seq- encoder-decoder models with attention. Given the Architecture: The model was absolutely based upon the
unavailability of resources for such task in Turkish, the attention-driven LSTM models and located with Azure
authors constructed two large Turkish MWP datasets by Cognitive Services in such a way that it lets developers put
translating established English datasets. The experiments translation together into their applications.
conducted with a variety of embedding models (BERT, Advantages: Great quality, and language-agnostic
Word2Vec) and neural networks (LSTM, GRU) showed that translation that is very easy to integrate into the Microsoft
the model with the highest precision attained an accuracy rate ecosystem.
of 72%. This study sets a benchmark for the future research
Disadvantages: As with most Seq2Seq models, this model
on Turkish MWPs and demonstrates that neural networks are
capable of dealing with complicated language-specific doesn't learn long-range dependencies and was quickly
problems in Turkish. [12]. replaced by Transformer-based systems.
4. Marian NMT smoother, rather more coherent translation, rather than
Overview: Marian NMT is a really efficient framework getting "stuck" on certain patterns.
developed by the Modern MT project. It is especially
optimized for sequence-to-sequence tasks, including 2. Vocab Size Limitation and Effective Tokenization
translation, with attention-based models. Background: Traditional models by and large rely on a
Architecture: Marian initially made use of RNN-based significantly larger vocab sizes without strict limits,
Seq2Seq models with attention but recently supported
implying that maybe higher memory requirement and
Transformers. Marian concentrates on the speed and
reduced speed both at training as well as inference.
efficiency by making use of a custom GPU implementation
to improve the speed of translation. Vocabulary Limited: the vocabulary is restricted to be only
Pros: Marian's very highly optimized Seq2Seq for faster the 20k most frequent tokens; therefore, the number of
translation and less utilization of resources makes it aptly unique tokens that get processed is limited, though rare
suitable for real-time applications. words and out-of-vocabulary words map to some generic
Cons: Though optimized, RNN-based Seq2Seq models placeholder; finally, special tokens such as <PAD> and
within Marian NMT suffered exactly the same limitations <START> are used to mark sequences start and padding.
other attention-based systems had, especially concerning Optimized Tokenization: the tokenizer is optimized for the
long sentences and intricate linguistic structures. frequency of tokens that come up in practice so that there is
a consistency in sequence length, achieved through padding,
5. IBM Watson Language Translator in mini-batches. This leads to better memory efficiency and
Overview: The initial versions of IBM Watson Language faster processing.
Translator too are by Seq2Seq with attention for translation Benefits:It reduces usage of memory. It focuses on frequent
tasks before they evolved into Transformer models. tokens. The system is less memory-intensive and thus
Architecture: Watson arrived with LSTM-based Seq2Seq optimized for use in low computational power devices.
models with attention, integrated with IBM's suite of AI It improves the inference speed: With a restricted
tools, hosted in the IBM cloud. vocabulary, so also does the dimension of the embedding
Advantages: of Seq2Seq with attention: So, the model has matrix; hence, lookups are faster. This property can be very
brought greater translated output quality compared to valuable for real-time applications.
previous technical domains particularly in terms of context
awareness and domain-dependent vocabulary. 3. Real-time Translation Readiness
Disadvantages: Even though Watson is much better than Background: Generally, the standard Seq2Seq models tend
the rule-based systems in terms of translation quality as well to be designed with batch-processing approaches and likely
as context treatment, it still isn't available and scalable will not be optimized for real-time responding required for
enough in comparison to Transformer models. such applications demanding immediate translations.
Design for Real-Time Performance: The system proposed
IV. PROPOSED SYSTEM will be designed taking into consideration real-time
translation, whereby both vocabulary and the tokenization
1. Scheduled Sampling for Training Background: techniques ought to be efficient, thereby accommodating
Traditional Seq2Seq models are trained with teacher streamlined batch processing along with single-sequence
forcing—a technique in which the ground-truth sequence inference that would help in generating the response time
used at each decoding step during training. Though effective quite quickly.
for fast training, teacher forcing suffers from an exposure Direct Single-Sentence Modeling Otherwise, model
bias: the model learns to depend too heavily on ground-truth optimization is through direct processing of single
sequences and cannot generalize well during inference when sentences, so that no heavy loading will degrade response
it has to predict from its own computations. time in the traditional models.
Scheduled Sampling: This is addressed by the proposed Benefits: Improved User Experience Directly translated for
system through scheduled sampling-a method in which, over no delay by quick autoresponders. Fits most interactive
epochs, the model gradually begins using its own previous applications such as a chat or even real-time conversation
predictions instead of the true previous work. The model translation.
transitions phase by phase across epochs so that the model Reduced Latency There will be no waiting, and with the
can build up its immunity to cumulative errors. handling of single sentences in translation, one would
Benefits: it improves the strength of the model: It actually escape delays.
trains to be such that it could depend on predictions more,
hence it is more resilient and better for real-life scenarios
where exact translations cannot be found.
Improves the Accuracy of Translation: Further, for the
diminishing exposure bias, the model is able to produce
 upon certain evaluation data, thanks to the attention
mechanism that allows the decoder to take relevant encoder
outputs when generating each word, thus retaining some level
of context, especially in longer sequences. The final model is
able to effectively translate from english to hindi, ensure that
meanings are preserved intact, and account for differences in
grammatical constructs. Other techniques, such as teacher
forcing and scheduled sampling, give the model more
robustness to the learning procedures by allowing the model
to discover correct predictions in their beginning stage and
Fig 1.1 Machine Translation Flow Chart
then slowly decrease dependability on labeled sequences.
Results show that seq2seq with attention can serve as an
effective architecture for NMT applications while still
V. METHODOLOGY yielding trustworthy performance under low-resource
circumstances. Nevertheless, the approach suggests how
It describes a systematic methodology to build and train a seq2seq with attention manages to achieve context-aware
neural machine translation model in accordance with the translation well, possibly adaptable to other language pairs
with other similar NMT issues.
architecture of seq2seq with an attention mechanism. The
first step was to load and preprocess the data. Here, the
english-hindi parallel text was tokenized, indexed, and VII. CONCLUSION
padded to obtain fixed-length input sequences. Each word
The seq2seq model with attention offers an effective
from the vocabulary mapping was given an integer index,
solution to the challenges of neural machine translation
while vocabulary mappings of both the english and hindi in common problems, such as dealing with complex
datasets were obtained from these original datasets. The sentence structures and low-resource languages. The
seq2seq comprises an encoder and a decoder. Both the mechanism of attention enables deeper contextual
encoder and decoder are thought to have embedding layers, understanding by aligning each output word with specific
which map words onto dense vector representations, and only segments of the input sequence, which leads to a more
then allows the model to capture the semantic relationships accurate translation. In this implementation, seq2seq with
between the words. The LSTM layer in the encoder was attention proved helpful in capturing subtleties and
assumed to encode the input sequence into its hidden states, syntactic structures necessary for reliable translation
encapsulating the contextual information across the time from english to hindi. Scheduled sampling with gradual
steps. This contextual information feeds the attention- introduction enhances the model's training optimally by
augmented decoder while generating the target hindi balancing the usage of true labels with that of model
predictions for high accuracy without over-reliance on
translation. The decoder computes the weights dynamically
ground truth labels. This work demonstrates that the
and pays attention to all positions of input so that it can lay attention-enabled seq2seq model does bring a significant
emphasis on the more relevant parts of a source sentence. quality improvement over translation, hence making this
Crossentropy was used as the loss function during the training model useful in low-resource or morphologically
process in teacher forcing conditions, so the number of inputs complex languages. Future work could push forward this
under teacher forcing was progressively reduced, making the architecture by testing transformer-based attention layers
model rely on the produced outputs. An optimizer called as well as more multilingual data for generalization. This
adam is available to finetune the parameters of the model. achievement highlights the need to focus attention within
Finally, after certain numbers of evaluation steps and training the NMT framework, which in turn can affect the
for multiple epochs, one can determine the translation development of other advanced NMT systems for other
accuracy, therefore providing an opportunity for assessing language pairs and resource conditions.
the correctness of the model. Such systematic technique helps
the model learn context-specific translations and adapt to VIII. REFERENCES
variations in sentence length and structure, hence rendering
the output translations of higher quality.
IEEE Citation: A. Slim, A. Melouah, U. Faghihi, and K.
Sahib, "Improving Neural Machine Translation for Low
Resource Algerian Dialect by Transductive Transfer
VI. RESULT
Learning Strategy," Arabian Journal for Science and
The seq2seq attention model has drastically improved Engineering, vol. 47, pp. 10411–10418, 2022, doi:
the quality of translation with long and complex syntactic 10.1007/s13369-022-06588-w. [1]
forms. It substantially enables a dynamic focus adjustment on
the relevant sections of the source sentence, thus significantly IEEE Citation: J. Lin, X. Sun, X. Ren, M. Li, and Q. Su,
enhancing the ability of the context to be translated. During "Learning When to Concentrate or Divert Attention:
training, the model shows lessened loss with increasing Self-Adaptive Attention Temperature for Neural
performance, as manifested through BLEU scores validated Machine Translation," in Proc. of the 2018 Conference
on Empirical Methods in Natural Language
Processing, Brussels, Belgium, Oct. 2018, pp. 2985- IEEE Citation: J. He et al., "Molecular optimization by
2990. [2] capturing chemist’s intuition using deep neural
networks," J. Cheminform., vol. 13, no. 26, pp. 1-17,
IEEE Citation: J. Huang, Y. Sun, W. Zhang, H. Wang, 2021, doi: 10.1186/s13321-021-00497-0.[11]
and T. Liu, "Entity Highlight Generation as Statistical
and Neural Machine Translation," IEEE/ACM IEEE Citation: E. Gedik and T. Güngör, "Solving
Transactions on Audio, Speech, and Language Turkish math word problems by sequence-to-sequence
Processing, vol. 26, no. 10, pp. 1860-1871, Oct. 2018, encoder-decoder models," Turkish Journal of Electrical
doi: 10.1109/TASLP.2018.2845111. [3] Engineering and Computer Sciences, vol. 31, no. 2, pp.
431–447, Mar. 2023, doi: 10.55730/1300-
IEEE Citation: R. Y. Benmalek, M. Khabsa, S. Desu, C. 0632.3993.[12]
Cardie, and M. Banko, "Keeping Notes: Conditional
Natural Language Generation with a Scratchpad
Mechanism," Proceedings of the 57th Annual Meeting
of the Association for Computational Linguistics,
Florence, Italy, 2019, pp. 4157-4167. [4]

IEEE Citation: A. Bapna, M. X. Chen, O. Firat, Y. Cao,

and Y. Wu, "Training Deeper Neural Machine
Translation Models with Transparent Attention,"
Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing, Brussels,
Belgium, 2018, pp. 3028-3033. [5]

IEEE Citation: Y.-H. Lee, D.-W. Jang, J.-B. Kim, R.-H.

Park, and H.-M. Park, "Audio–Visual Speech
Recognition Based on Dual Cross-Modality Attentions
with the Transformer Model," Applied Sciences, vol.
10, no. 20, pp. 7263, 2020. [6]

IEEE Citation: P. Colombo, E. Chapuis, M. Manica, E.

Vignon, G. Varni, and C. Clavel, "Guiding Attention in
Sequence-to-Sequence Models for Dialogue Act
Prediction," Proc. Thirty-Fourth AAAI Conf. Artificial
Intelligence (AAAI-20), 2020, pp. 7594-7601. [7]

IEEE Citation: Y. Li, J. Li, J. Jiang, S. Tao, H. Yang,

and M. Zhang, "P-Transformer: Towards Better
Document-to-Document Neural Machine Translation,"
IEEE/ACM Transactions on Audio, Speech, and
Language Processing, vol. 31, pp. 3859–3870, 2023.
[8]

IEEE Citation: C. Lu, "Performance analysis of

attention mechanism and teacher forcing ratio based on
machine translation," Journal of Physics: Conference
Series, vol. 2580, p. 012006, 2023, doi: 10.1088/1742-
6596/2580/1/012006. [9]

IEEE Citation: M. M. Al-Sayed, "Workload Time

Series Cumulative Prediction Mechanism for Cloud
Resources Using Neural Machine Translation
Technique," J. Grid Computing, vol. 20, no. 16, pp. 1-
29, May 2022.[10]