Tim Rocktäschel

🚀 I’m thrilled to share that our paper “STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data” has been accepted at NeurIPS 2025, San Diego, 2-7 Dec 2025! 🎉 Our contribution: https://lnkd.in/eHcQb6a6 This work is the product of a collaboration between University of Glasgow School of Computing Science (Knowledge & Data Engineering Systems #KDES) and the BMW Group. Huge thanks to my co-authors & collaborators: Maximilian Forstenhäusler, Daniel Külzer, Shameem A Puthiya Parambath, and Dr. Natascha Weber (Harth). 🔍 What problem we tackled Real-world sequential (e.g. spatiotemporal / time series) data often poses serious challenges: --Non-stationarity: signal properties changing over time. --Irregular sampling (missing data / varying intervals). --Need for latent representations that are task-informed, i.e. tuned to downstream classification / anomaly detection tasks, not just generic embeddings. 🎯 STaRFormer Contribution We propose STaRFormer, a framework that combines two main ideas: [*] Dynamic Attention-Based Regional Masking by helping the model learn robustness to irregularities (missing/unreliable/malformed parts) and focus on informative regions. [*] Semi-Supervised Contrastive Learning, where STaRFormer instead of treating labeled and unlabeled data separately, it blends them through a task-aware contrastive learning objective. 💡 Why this matters --Better representations: STaRFormer yields embeddings that are more useful for downstream tasks. --General framework: STaRFormer handles regular, non-stationary, irregular-sampling, sequence-level and element-wise tasks—so less need for highly tailored models per scenario. NeurIPS 2025 Record: https://lnkd.in/eB6YzG26 #NeurIPS2025 #MachineLearning #RepresentationLearning #AIResearch #DeepLearning #TimeSeries #KDES

111

15 Comments

A bit late to the party, but happy to share our #NeurIPS2024 paper "Understanding Information Storage and Transfer in Multi-modal Large Language Models": https://lnkd.in/dq4FD5mj! We use a causal tracing methodology to study how multi-modal LLMs (MLLMs) like LLaVa and Phi retrieve information required to answer factual visual questions. Our experiments reveal that 1) MLLMs rely on earlier layers for information storage compared to LLMs, and 2) a small subset of visual tokens play a crucial role in transferring visual information to these causal blocks. We also introduce a new model-editing algorithm that leverages these findings to correct errors and insert new information into MLLMs by targeting these early causal blocks. You can find the code for our work here: https://lnkd.in/d4gr7fFB Excellent work led by Samyadeep Basu, in collaboration with Besmira Nushi, Cecily Morrison, Martin Grayson, and Soheil Feizi.

191

5 Comments

Missing data is ubiquitous in modern socio-economic systems and in digital platforms, and it’s often missing not at random (MNAR): users with extreme experiences are more likely to leave feedback. This can bias population estimates and hurt decision-making. In our recent paper, "Partial Identification under Missing Data Using Weak Shadow Variables from Pretrained Models," written with my PhD student Hongyu Chen and Ruoxuan Xiong, we ask: How can we estimate population quantities (e.g., mean outcomes) under MNAR? And can modern AI models help? We view LLM predictions as weak shadow variables because of three key features: (1) they don’t directly affect the missingness mechanism; (2) they can approximate human judgments; and (3) they’re imperfect, so we shouldn’t treat them as replacements for ground truth. Key idea: We introduce a partial identification framework where sharp bounds are obtained by solving a pair of constrained linear programs. LLM predictions naturally enter as additional constraints, tightening the identification set, without requiring perfect predictions. In experiments on customer-service dialogues, LLM predictions are often ill-conditioned for classical point-identification approaches. However, they remain highly effective in our framework, shrinking identification intervals by 75–83%. You can read our paper on arXiv: https://lnkd.in/eax3SBk8

140

4 Comments

🚨 Exciting New Paper Alert! Our latest research dives deep into how large language models (LLMs), specifically Llama3-70B, achieve abstract reasoning through emergent symbolic mechanisms: 1️⃣ Symbol Abstraction Heads: Early network layers transform input tokens into abstract symbolic variables based on their relational structures. 2️⃣ Symbolic Induction Heads: Intermediate layers then perform symbolic sequence induction, reasoning over these abstracted variables. 3️⃣ Retrieval Heads: Later layers predict subsequent tokens by retrieving values tied to previously established abstract variables. This work provides crucial insights into how neural networks begin to develop symbolic reasoning capabilities, bridging the gap between neural computation and symbolic logic. Neural networks are not just pattern matchers—they're starting to "think" symbolically! 🧠🔍 Check out the full paper here: https://lnkd.in/gSVn8PTC #LLMs #AI #SymbolicAI #MachineLearning #Research

252

12 Comments

My last post about NeurIPS allowing research with hallucinated citations struck a nerve. Now, ICLR'26 has responded by #deskrejecting any paper with LLM-hallucinated references. Bold move, right? But does it actually solve the problem? The questions become: 🚩 How do you reliably verify thousands of citations across thousands of submissions? ⁉️ Manually? That's not scalable. ⁉️ Automatically? The same AI systems causing the problem would need to help solve it. 🤔 🚩 And if desk-rejection becomes the standard across conferences, what happens next? ⁉️ Do we penalize researchers who "unknowingly" used AI tools that hallucinated references, or ⁉️ Maybe we should Start Building citation verification systems before enforcing these policies? 🤷‍♀️ Even worse.... 🛑 Does this create a gap where well-resourced and funded labs catch errors before submission, while smaller researchers, especially those in the Global South with fewer resources, get desk-rejected? 🛑 Detecting hallucinated citations is harder than it sounds. 🤨 What do you think? Is this the right step, or are we getting ahead of ourselves? #Phd #publication #research #useAIwisely #AI #Hallucination #Conferences #contentmoderation #ICLR2026 #AcademicIntegrity

154

29 Comments

🚨 New paper on the limits of identifiability theory! Mathematical guarantees for neural representations are thrilling—especially in a field driven by trial and error. But sometimes theory lives in "fairyland" ... 👉 arxiv.org/abs/2504.13101 In Theory fairyland: Infinite data, models converge globally, latents on the right manifold, with the right statistics... 🧚 In Practice: Noise, Batch size, learning rate, data augmentations, inductive biases, did I mention noise? & all the gritty stuff that actually matters 🛠️ (We still 💙 theory.) So—does this mean theory is doomed, and AI engineering is just a random walk? Not at all! 💡 Patrik Reizinger Wieland Brendel Randall Balestriero did an amazing job clarifying where theory can help practice—and where practice should inspire theory. 🤝 ➕ Bonus: Theory can explain the “Platonic Representation Hypothesis” (https://lnkd.in/eXyY6PGU striking observation that different models often learn the same representations. With the right assumptions (hopefully not too 🧚‍♀️), you get a rigorous mathematical explanation for why this happens 🤓

73

New paper! Available benchmarks for long-context reasoning in Large Language Models do not distinguish the effect of task complexity, presence of irrelevant information, and task length. @daniel Kaiser invented CogniLoad, a synthetic benchmark grounded in Cognitive Load Theory. CogniLoad is used to test 22 reasoning LLMs, and reveals distinct performance sensitivities, identifying task length as a dominant factor, varied tolerances to intrinsic complexity and a never seen before U-shaped response to presence of distracting information. Take a look! https://lnkd.in/dVwAST9B Ali Ramezani-Kebrya, Benjamin Ricaud, Integreat - Norwegian Centre for Knowledge-driven Machine Learning, University of Oslo, UiT- The Arctic University of Norway

59

1 Comment