Style, Semantics, and Other
Things
Krishnapriya Vishnubhotla (KP)
Intro: Style in NLP
● Uniqueness of writing style
● Due to:
○ Lexical choices (big words vs small words)
○ Sentence structure (short n simple vs complex with clauses)
● Stylometry:
○ Surface features (word lengths, sentence lengths)
○ Lexical features (LIWC, number of hapax legomena)
○ Syntactic features (function word frequencies, PoS tag frequencies, parse tree features, character trigrams)
● Authorship attribution, plagiarism detection, digital forensics
Form and Meaning
● Text generation process:
○ a meaning, or content +
○ Form, or style
● Multiple surface realisations are possible for the same meaning
● Natural language corpora:
○ Complex vs simple wikipedia
○ Literary translations
● Closely related to: paraphrases
Paraphrases
● Paraphrase identification, generation
● Datasets: Quora Question Pairs, Microsoft Research Paraphrase Corpus, ParaNMT
● Semantic Textual Similarity tasks
NLP: Style Transfer
● Lots of work on style transfer in NLP
● “Style” ---> factor of variation
○ Sentiment
○ Attributes
○ Topics
● Usually guided by the dataset used.
● Problematic:
○ What should be preserved?
○ Adds to already problematic evaluation metrics
Complications
● There are no true synonyms -- “near-synonyms”
● Changing active to passive → change of focus
● Pragmatics -- viewpoint, framing, denotation, connotation, implication.
● Can draw some fuzzy boundaries between clusters of near-synonyms at a word-level
○ What about for phrases/sentences/documents?
● Style: Literary definition: what is “lost in translation”
Meaning Representations
● Formal representation of meaning/semantics
● Lots of CL research on logical forms, compositionality
● Two relatively-recent projects I came across
○ Abstract Meaning Representation (AMR)
○ Minimal Recursion Semantics (MRS)
Abstract Meaning Representation
● Rooted, directed, (edge+leaf)-labelled graph
● Uses PropBank frames
● Example: “The dog is eating a bone,”
Relations
Variable / Concept
● “The dog ate the bone that he found.”
● Has ways to handle:
○ Coreference
○ Negation
○ Numbers/quantity
○ Names
Generalisation capabilities
- The man described the mission as a disaster.
- The man’s description of the mission: disaster. Same AMR.
- As the man described it, the mission was a disaster.
- The man described the mission as disastrous.
● Abstracts away morphological and syntactic variations.
● But does not handle synonyms
○ “afraid” and “terrified” are treated as different concepts.
● Useful?
○ Not yet.
○ Purpose: dataset to help develop algorithms that can generate AMRs.
Minimal Recursion Semantics
● Another formalism: phrase structure grammar
● More fine-grained
● Can distinguish between tense, number.
● Practical utility:
○ Has a command-line parser you can use
○ Can generate simple paraphrases
Practical Utility
● Unlikely that they can parse many real-world sentences:
○ LIT paper: successful at 19.7% of SNLI sentences
● Using AMR to detect paraphrases:
○ ~85% on the Microsoft Paraphrase Corpus
● A separate research problem, not a tool to be used.
Back to Representation Learning
● Let us assume we have…
● Some proxy information for:
○ Form
○ Meaning Text t
Form Vector Meaning Vector
Stylistic similarity Semantic similarity
Neural Models
z classifier
● Modified Autoencoders
Paraphrases
● Encode into two vectors
● Use both to reconstruct
● Restrict information using
motivational/adversarial
discriminators
Semantic z classifier
Syntactic
What kinds of supervision?
Datasets
● Style class labels
● Paraphrases ● Paraphrase datasets
● Heuristic info: ● Parallel style transfer datasets
○ BoW for content ○ Formality
● Syntax: Syntax tree features ○ Diachronic language change
○ Tree edit distance ● Data-to-text datasets
○ ~Synthetic
Synthetic Dataset: PersonageNLG
● Personality model might be questionable
● BUT gives us two neat dimensions of variation.
All the losses later….
Evaluation:
● Style transfer (swap variables + generate)
● Retrieval
● Prediction (kNN)
More supervision == better representations
● Kinda boring
● Just train a separate supervised model
for each end-goal?
● Style transfer:
○ Generation problems
○ Evaluation problems
● Real-world text: not so cleanly
separable.
:(
What would be interesting?
● Unsupervised disentanglement?
○ beta-VAE in vision
○ At least for the synthetic dataset
● Evaluating the representations:
○ Probe for linguistic knowledge/features
○ Robust to “noise”? → domain adaptation/zero-shot prediction
● Using pre-trained models?
● (TBD) Should the latent spaces be entirely unrelated?
○ Where do style and semantics intersect?
○ What is a “latent space of sentences” anyway?
