## Problem LLMs take an input, and give you a statistically likely output given it's training data and bunch of other parameters. However, in the current paradigm, it makes no attempt to understand what the user knows. A simple RAG strategy that keeps the users' responses in a vector database partially remediates this, by injecting related conversations from the past into the context for each chat. However, to optimize for learning, using the results of a fuzzy search over past conversations is insufficient to determine how much a user understands, their motivations and their mental model. The role of a good tutor is to identify incorrect mental models and correct them, and we'd like AI to get better at that. Essentially, the problem here is that there is a body of knowledge or skill that a user wants to learn. The (intelligent) tutor's goal is to maximize the learning for the user. Simply dumping the information to the user is a suboptimal experience for the average learner. Instead, we have to optimize for the following: 1. **Recall**: a learner must be able to recollect what they learnt, and memory is consolidated by active recall of subject matter across a spread out revision schedule. Passively consuming information (reading, watching, listening) is ineffective for remembering things well. 2. **Synthesis & Application**: as mentioned above, active recall (i.e. 'remembering from scratch') is essential to remembering what is learnt, but, in addition to that, synthesizing new information or artifacts (writing, speech, projects or even a mind map) integrates the knowledge into practice in a way that it can over time become instinctive and intuitive, allowing the learner to apply it in real-life scenarios. 3. **Correcting misconceptions**: different learners start the learning process with different existing mental models, and the process of learning involves either changing that mental model or accommodating new facts into it (accommodating vs assimilating). Research indicates that even the most 'engaging' learning content often leads learners to simply affirm existing misunderstandings or biases unless they are made to notice an error in their mental model, activating the limbic friction and neuroplasticity for accommodating or assimilating new information. 4. **Motivation**: to truly accelerate the learning, the user must see the relevance of content to them or their interests. This is different across all learners, and a good tutor often leverages the personal relationship they have with a learner to understand what might be interesting to them or what their eventual goals are, tailoring both content and learning styles to that. ## Research Questions There are a few research questions here: 1. For a given concept or subject, how can we represent: 1. the 'correct' or desirable mental models for knowledge and skills? 2. the learner's mental model while they are learning it? 3. the learner's memory of that mental model over time? 4. the learner's motivation(s) for learning it? 5. the desirable learning path(s) (i.e. sequencing of lessons)? 2. How can the system infer the learner's mental models via conversation? 3. How can the system diff: 1. the **learner's mental model** with the **desirable concept model**? 2. the **learner's motivation model** with a desired learning path? 4. Based on the diffs of the models, how can the system suggest: 1. learning exercises that reveal the mismatch in understanding to the learner ('perceptual learning modules') 2. content and exercises tailored to the user's motivations? ## Potential Solutions We look at the following representations. 1. **Hierarchical lists (simplest structural representation)** 2. **Knowledge graphs (richer structural representation)** 3. **Bayesian concept learning (reasoning over hypotheses)** 4. **Bayesian knowledge tracing (tracking mastery over time)** The first to cover **structure of knowledge**, while the other ones cover **inference about meaning** and **inference about mastery**. ### 1. Good ol' lists The simplest way to represent understanding is as a **hierarchical list of concepts**. Instead of showing relationships explicitly, this structure assumes that concepts can be organized into parent–child relationships: broad ideas at the top, increasingly specific sub-ideas underneath. Hierarchical models of curriculum design implicitly rely on this principle: understanding is not just knowing items but knowing _how they nest_ within larger explanatory frames. This format is intuitive, compact, and easy to operationalize (e.g. textbooks, syllabi). A learner’s current understanding can be represented the same way: a partially filled-out hierarchy, where some branches are deeply expanded (expertise), others are shallow (familiarity), and some are empty (unlearned). But hierarchies are a simplification. They capture _scope_, but almost none of the nuance of real conceptual structure. Learning scientists like [Chi (2005)](https://www.education.umd.edu/uploads/6H/7u/6H7uvYFlf_Chi2005.pdf) have shown that many misconceptions arise not from missing items in a hierarchy but from incorrect relational assumptions, i.e. things hierarchies cannot encode. **Limitations:** - Assumes knowledge is tree-shaped, not networked - Cannot represent misconceptions or conflicting models - No representation of uncertainty, strength, or conditional activation - Implies linear progression even when learning is nonlinear ### 2. Knowledge Graphs A simple way to improve on simple lists and externalize understanding is to treat knowledge as a graph, i.e. - Concepts are nodes (e.g., _Voltage_, _Current_, _Resistance_, _Ohm’s Law_) - Relationships are edges (e.g., _Voltage depends on Current and Resistance_, _Current flows when there is a Voltage difference_, _Resistance reduces Current_) In this framing, we can maintain two parallel graphs: an expert graph and a learner graph. The **expert graph** represents the intended mental model for a topic, i.e. the structure we hope a learner eventually internalizes. This idea goes back to [Donald's Norman’s work on mental models](https://ar264sweeney.wordpress.com/wp-content/uploads/2015/11/norman_mentalmodels.pdf), which describes how expertise depends not just on facts but on how they relate and constrain each other. Later work on [**concept maps** by Novak & Cañas](https://cmap.ihmc.us/docs/theory-of-concept-maps) operationalized this into explicit, editable external knowledge structures: a graph becomes an object you can inspect, critique, annotate, and refine. The **learner graph**, by contrast, reflects _where the learner currently is_. At the beginning it’s sparse and tentative: a few nodes, weak links, and maybe a misleading shortcut or two. Chi’s research on misconceptions shows that many student errors aren’t about missing nodes but about **wrong edges**, i.e. relationships don’t actually hold in the expert model. As learning progresses, the learner graph becomes denser and more hierarchical; edges are pruned or reorganized, and the structure becomes more isomorphic to the expert version. Viewed this way, learning isn’t the accumulation of facts, but the **progressive restructuring of a graph**. So knowledge graphs give us a direct and inspectable representation of understanding: what concepts exist, how are they related, how closely does the learner’s structure map to the expert’s They’re simple, visual, and interpretable, which is why they’ve shown up repeatedly in instructional design, cognitive tutoring systems, and concept inventory research. **Limitations of knowledge graphs:** - They represent _what is connected_, but not **how strongly** or **with what confidence** those connections are held. - They assume a single stable structure, while learners often hold **multiple, competing mental models** that activate in different contexts. - They capture relationships but not the **inference process**, i.e. how learners generate hypotheses, update beliefs, or generalize from examples. - They struggle to encode **temporal dynamics** (how understanding evolves trial-by-trial), unless additional layers or metadata are added. - They treat misconceptions as incorrect edges, but do not provide a natural mechanism for explaining **why that incorrect structure is compelling** or how it might change. - They are primarily static and descriptive rather than **predictive**: knowing the shape of the graph doesn’t tell us how a learner will behave on a new task or respond to new evidence. ## 3. Probabilistic Models of Congitions Josh Tenenbaum’s [work](https://home.cs.colorado.edu/~mozer/Teaching/syllabi/ProbabilisticModels/readings/Tenenbaum1999.pdf?utm_source=chatgpt.com) is the classic starting point. Instead of committing to a single representation (like a list or graph), these models treat learning as **Bayesian inference over a space of hypotheses**: a learner entertains many possible explanations and updates their beliefs as evidence arrives. In Tenenbaum’s classic experiments, learners infer the boundaries of a concept such as a geometric region (e.g., _all possible rectangles consistent with the observed examples_) and update their internal hypothesis distribution based on Bayesian principles. Strikingly, these models replicate human generalization patterns, especially our ability to learn rapidly from very few examples, something most deterministic or structural models fail to explain. **Limitations:** - Requires defining the hypothesis space in advance, which may be difficult or unrealistic for rich domains. - Assumes the learner performs (or approximates) Bayesian reasoning, which may not reflect cognitive constraints or heuristics. - Often models idealized toy tasks more easily than messy, conceptual, real-world learning. - Captures inference and uncertainty well, but doesn’t fully address _how_ hypotheses are represented, stored, or transformed over longer learning trajectories. ## 4. Bayesian Knowledge Tracing In education and intelligent tutoring, people often mean something more operational: > Given a log of what a learner did (right/wrong answers, actions), infer how well they understand each skill. BKT is a hidden Markov model over a binary “knows/doesn’t know” state for each skill. After each interaction, you update the probability that the student has mastered that skill. Bayesian Knowledge Tracing (BKT) shifts the focus from _representing concepts_ to _tracking mastery over time_. Instead of modeling entire concept spaces or hypothesis distributions, BKT treats each skill as a hidden binary state: _known_ or _not yet known_. After each learner action (e.g., answering a question), the system updates its belief about whether the learner has mastered that skill using Bayesian inference. The model includes parameters for **guess** (a correct answer despite not knowing), **slip** (an incorrect answer despite knowing), **learning rate**, and **prior knowledge**. This makes BKT useful in adaptive tutoring systems and learning platforms, where the goal is not to explain how concepts are mentally represented but to estimate whether a learner is ready to advance. Because it operates incrementally and probabilistically, BKT can personalize learning sequences, identify plateaus, and trigger targeted interventions. # Past Work in Learning & Education #### Knowledge Graphs: - [Learning With Concept and Knowledge Maps: A Meta-Analysis by John C. Nesbit and Olusola O. Adesope](https://pdfs.semanticscholar.org/f6e0/72d8e57da9806c517a5baa9c44507cb3ae81.pdf?skipShowableCheck=true&utm_source=chatgpt.com) - [Effects of concept mapping on L2 learning: a meta-analysis of studies from 2006 to 2024](https://link.springer.com/article/10.1186/s40862-025-00331-2) #### Bayesian Knowledge Tracing - [Cognitive Algebra II Tutor: Modeling Student Knowledge: Cognitive Tutors in High School and College](https://www.cs.ubc.ca/~conati/522/532b-2022/papers/Corbett2000_Article_ModelingStudentKnowledgeCognit.pdf?utm_source=chatgpt.com) - [The Sum is Greater than the Parts: Ensembling Models of Student Knowledge in Educational Software](https://www.kdd.org/exploration_files/V13-02-06-Pardos.pdf?utm_source=chatgpt.com) - [Knowledge Tracing Models’ Predictive Performance when a Student Starts a Skill](https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_126.pdf?utm_source=chatgpt.com) ## Open-Source Intelligent Tutoring / Adaptive Learning Projects - **Dodona**: intelligent tutoring for programming (feedback, analytics) - https://github.com/dodona-edu/dodona - https://papyros.dodona.be/ - **PyKT Toolkit**: implementations of knowledge-tracing models (BKT, DKT, GKT, AKT, etc.) - [https://github.com/pykt-team/pykt-toolkit](https://github.com/pykt-team/pykt-toolkit) - **Oppia**: adaptive learning platform with branching feedback and learner models - [https://github.com/oppia/oppia](https://github.com/oppia/oppia) - **OpenDSA**: step-based interactive CS exercises with feedback and sequencing - [https://github.com/OpenDSA/OpenDSA](https://github.com/OpenDSA/OpenDSA)