Competency Mastery Concurrency ADR

[jesperhodge]

UPDATE: ADRs in this PR have been completely rewritten as of July 17, 2026.

Strong involvement from Claude (AI).

I haven't fully reviewed changes to diagrams and ADRs 2 and 3 yet, made with Claude, need my own review still.

Summary

These are some significant changes to the data model and approach. They will warrant a good amount of discussion.
A main concern is that the leaf node table and leaf node history table could possibly contain billions of rows, at the scale of the PersistentSubsectionGrade table of edx-platform. The current solution accepts this, but is this acceptable for OpenEdx? If not, there is also a rejected alternative to not store leaf nodes and just compute them at read time, but that would make it harder to keep subsection statuses frozen so that later competency criteria changes don't change the status.

ADR 4: How learner competency mastery is recorded
correctly under concurrent, out-of-order grade-change events without
paying a per-event serialization cost at scale.

ADR 5: How competency criteria statuses for learners are stored at scale, given the tables
can be massive (billions of rows).

[openedx-webhooks]

Thanks for the pull request, @jesperhodge!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[mgwozdz-unicon]

Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.

I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.

Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.

Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.

[ormsbee]

Maybe this is a silly question, but can't we mitigate this by making sure to commit the transaction of the leaf statuses, and then re-read the latest commited state of the leaves before doing the roll-up?

[ormsbee]

Another thing we can do (if it's necessary) is to arrange it so that the roll-ups are a log that have an incrementing version number, and create a composite unique index that would cause a conflict that would reject concurrent writes. The losing write would have to catch the error, increment the version number again, and re-read the database state.

[jesperhodge]

I think this question is not relevant anymore with the current version, right? Since now we only process on one worker at a time.

[jesperhodge]

Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.

I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.

Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.

Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.

@mgwozdz-unicon I'll improve the description in the ADR.

Here's why this solution is not good:

It fights the batching that the performance depends on. The chosen design's throughput comes from batching across learners: one bulk read of current statuses, evaluate the whole batch in memory, one bulk write — a few round-trips regardless of how many learners/events are in the batch. A per-learner lock is naturally per-learner (in the earlier design, per-event): acquire lock → read that learner → evaluate → write → release, one learner at a time. That reintroduces per-learner (or per-event) transaction/commit overhead plus lock acquire/release churn, which is exactly the overhead batching was collapsing. Under bursty grading across many learners, that per-unit overhead dominates.

Lock-lifecycle machinery would be multiplied across millions of keys. One deployment-wide lock has exactly one lifecycle to run: acquisition, timeout, stale-lock recovery on a crashed worker. A per-learner lock needs a per-learner lock table (or keyed advisory locks) and that same lifecycle replicated across potentially millions of learner keys, held and waited on concurrently, plus a connection+worker tied up per contended lock.

[mgwozdz-unicon]

Either it's not clear to me what this is trying to say or it contradicts point 3 above it. I think my confusion might stem from the phrase "with the grade" because that is implying to me that writes to our competency status tables roll up atomically with the subsection grade table. But I think this is trying to say that writes for the competency criteria hierarchy status tables commit atomically with each other. Am I confused or does this need to be reworded for clarity, or both?

[mgwozdz-unicon]

Nevermind, I reread point 3, and I see that they're not contradictory now. It's possible that this could still be made clearer though since I did have to reread to understand.

[mgwozdz-unicon]

I think it would be nice if the Rejected Alternatives each had brief Pros and Cons.

[mgwozdz-unicon]

Or a rationale sentence or two like in ADR 5 below.

[mgwozdz-unicon]

This numbering is off

[mgwozdz-unicon]

@jesperhodge The inline comments are from my completed review. I had Claude review as well. These are the things that came up from that which I found to be relevant:

Questions

1. docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst (mechanism 2, "When a child advances, its parent is recomputed..."): the parent-recompute lock (SELECT ... FOR UPDATE) only guarantees a fresh read of sibling children if it's the first read in that DB transaction. MySQL's default isolation, REPEATABLE READ, fixes a transaction's read snapshot at its first plain (non-locking) SELECT. A locking read like FOR UPDATE always sees latest-committed data and doesn't itself fix the snapshot, but any earlier plain read in the same transaction (e.g. re-reading the grade or the leaf row before the parent lock) would. If the recorder does any plain read before taking the parent lock, the later child reads could still be served from that earlier snapshot rather than post-lock data, silently reintroducing the write-skew this mechanism is meant to prevent. Nothing in the ADR states the required read order or assumed isolation level. Worth pinning down explicitly, e.g. "the parent lock must be the transaction's first read" or "isolation is set to READ COMMITTED for this path."

2. docs/openedx_learning/decisions/0005-competency-mastery-storage.rst ("Enable read-replica offload for heavy reads for the leaf tables"): read-replica offload is scoped only to the leaf-level tables (StudentCompetencyCriteriaStatus and StudentCompetencyCriteriaStatusHistory). Dashboards presumably also read group- and top-level status. Is leaving those two levels off the replica path intentional (they're orders of magnitude smaller, so primary-DB reads are cheap enough there), or an oversight? A one-line justification either way would help.

I'm pretty sure that this was because those tables are smaller, but I agree that this should be made clearer.

Suggestion

docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst (mechanism 5, "The leaf HISTORY append runs outside the transaction, best effort"): the leaf HISTORY append is explicitly best-effort and non-blocking after the transaction commits, with no stated retry or reconciliation path. Since HISTORY exists specifically to serve as an audit trail (per ADR 0005's context section), a silently dropped append is a permanent audit gap. Worth at least a sentence on the acceptable failure mode: logged and ignored, or retried.

I would prefer at least one retry before logging and ignoring. If there's an accepted pattern in openedx for this though, then I defer to that.

Nits

• docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst: typo, "acvance" should be "advance."
• docs/openedx_learning/decisions/0002-competency-criteria-model.rst: list items 7, 9, and 11 (the new History-table index entries) have two spaces after the number, while the rest of the list uses one space. Doesn't break the Sphinx build, just inconsistent.
• docs/openedx_learning/decisions/0002-competency-criteria-model.rst: double blank line before the "Intended update flow" section.

[ormsbee]

Django's design assumes READ-COMMITTED, and now defaults to that on MySQL unless it's explicitly overridden in the connection options.

In [3]: from django.db import connection
   ...: 
   ...: with connection.cursor() as cursor:
   ...:     cursor.execute("SELECT @@transaction_isolation;")
   ...:     print(cursor.fetchone()[0])
   ...: 
READ-COMMITTED

Running in higher isolation levels is not supported by our platform.

[jesperhodge]

@mgwozdz-unicon @ormsbee I updated the ADRs based on your comments.
Main change: moved the leaf HISTORY append out of the grade transaction into its own transaction.on_commit-dispatched, retrying Celery task, and added a unique constraint on the advance (learner, node, status) as its idempotency key.

[mgwozdz-unicon]

Thanks for getting those changes in! Approving, though Claude flagged one more thing: the db_constraint=False decision itself has no stated rationale anywhere, it's the one bullet in ADR 0005 without any, while everything else in that doc (64-bit PKs, the dedicated HISTORY alias, read-replica scoping) explains why. This is the same field Braden raised a deletion-protection concern about on #661: that's resolved since his approach keys off the competency_criteria_id FK, not user_id, but that resolution isn't written down yet either. What's the actual reasoning for dropping the constraint on user_id here, and is it established for this table specifically, or assumed by analogy to StudentModule/PersistentSubsectionGrade? Not blocking on this, just think it might be good to add.

[ormsbee]

The concurrency ADR looks good to me. I have just a couple of concerns with the storage one. Will write up in the morning.

[ormsbee]

These concerns are over the data storage portion. Please forgive me if I've made these comments earlier. (I thought I had already commented on the 64-bit ints before, but maybe I never properly submitted that review. ☹️)

[ormsbee]

It does not block this ADR, but let's talk with Product/UX folks and explore just how much of an audit history we really need. History storage is the most dangerously unbounded thing in this system. Moving it to a separate system and potentially pushing it to another database entirely doesn't really solve it.

For instance, say we determine that we only need history showing when they cross the aggregate threshold for the first time. We could potentially save billions of rows of storage for a large site that would otherwise be storing a new row for every time someone clicked "submit" on a problem somewhere.

[jesperhodge]

@mgwozdz-unicon what are your thoughts?

[jesperhodge]

Yes, I think that is already the intention here, let me know what you think:
status advances are only monotonic, e.g. from partially demonstrated to demonstrated. So whenever there is no such status advance, we will not write anything into the history table.
I even gave it a hard db constraint in ADR 0002:
" 7. StudentCompetencyCriteriaStatusHistory(user_id, competency_criteria_id, status_id) (unique -- at most one HISTORY row per learner, leaf, and status level, which also serves as the idempotency key for the append in :ref:openedx-learning-adr-0004)"
I will make this a separate point so that it is more clear.
Think like this: The active table stores group and leaf nodes - for each learner for each subsection they attempt, for example. Sure, that is a lot.
Now when the learner tries a subsection multiple times and never crosses the grade threshold, it will remain "attempted, not demonstrated" and only one history entry will be written - because there is only a one-time status advance.
Since every node just has 3 or 4 statuses - for example "not demonstrated" -> "partially demonstrated" -> "demonstrated" for group nodes (I'm paraphrasing here, it might be a little inaccurate), the history table will only ever receive 3 or 4 writes for the group node.
So the history table will only ever be 3-4x as large as the active table.

What are your thoughts so far? Does that alleviate your concern, or do you see further opportunity to cut down the history table?

Also, one note: if we want to allow course authors or django admins to manually edit the status later downwards in case there was a mistake, and we want to keep that in the audit trail as well, we need to drop the unique constraint.

[mgwozdz-unicon]

I'm pretty sure some people will definitely want to see the history at that granular of a level, and in a few different use cases.

One of the things I remember from our interviews with institutions was that many folks were talking about the importance of learners being able to see their historical progress of competency attainment across multiple attempts in a single assignment as well as across the entire course, and surely at a program level as well.

But there will also be analytics/reporting use cases where instructors and instructional designers will want to understand some kind of aggregation of how learners were progressing through each material as it rolls up to the competency so that they know which learning materials need improvements and can try to get insights about which learning materials did well so that they can emulate that success.

But yes, the product folks and our CBE SME Jason could probably all provide more insights here. I'm sure that there are certain metrics that would take priority over others, but I feel pretty confident that auditing historical records will be desired at the leaf/subsection level, which is the one that is the biggest storage challenge. It's actually some of the middle roll-ups that some people may not care about as much, but I also think that some people will definitely want all the detail they can get.

If someone has a reasonably complicated logical hierarchy for how to demonstrate mastery of a particular competency and a learner inquires about why they haven't received their badge for it, then someone will need to do an audit to figure out where in that logical hierarchy the student didn't achieve so that they can explain this to them.

To Dave's point later on, there probably is a limit to the amount of time that a learner would ask the question "I thought I demonstrated mastery of this competency. Do an audit to find out why I was told I haven't." Which could probably make it safe to truncate the table at some point in history, though I know there are also laws and retention periods and such that need to be adhered to.

Out of these use cases, the one that would likely want historical data furthest in the past is actually probably the analytics/reporting case in the event of a course that runs over several months and they want to compare data for a specific subsection from the last 5-10 times this course took place, for example.

[ormsbee]

Suggested change

	of the table's size; the dashboard reads this feature performs are exactly such point lookups. What
	of the table's size; the dashboard that reads this performs exactly such point lookups. What

Is this what was meant?

[ormsbee]

Okay, the database router thing was a giant, ugly hack, because I threw together the original StudentModuleHistory model in a half-day hackathon project to help out one of the first CS courses on the platform, and I neglected to override the primary key defaults to make it 64-bit. The result was that we were running out of primary keys, and some devops folks (Kevin I think?) retrofitted a new table in there

Also, the courseware_studentmodulehistory table stays somewhat manageable for large instances because they truncate it from time to time, because this history is used for course staff debugging purposes, not auditing. So this week's history is really important when a student is asking, "I submitted the right answer, why is the grader marking me wrong?", but that's not a thing we need to look up six months from now.

[ormsbee]

This shouldn't need to be justified, as BigAutoField should be the default practice per OEP-38.

Also, please don't put up anything to imply that UnsignedBigIntAutoField is a good idea. It was an overkill reaction to having 32-bit primary keys run out on key tables, but it's more headache than it's worth. It theoretically gives us 2X the space of BigAutoField, but (a) the existing space of BigAutoField is so big that we'll never realistically reach it; (b) reaching it would break the database in a bunch of other ways anyway; and (c) declaring a custom field type like that introduces maintenance headaches like... (d) painful compatibility issues with PostgreSQL, which does not support unsigned ints.

[ormsbee]

Please don't do this. See my comment above about why this routing exists. Let's keep it simple and in the same database so that transactions just work.

[ormsbee]

I'm not clear as to what part of the referenced ADR this is drawing from, but our convention in this repo has been to have real foreign keys to the User model—it's just that we respect Django's ability for projects to define custom user models by grabbing it from settings.AUTH_USER_MODEL, instead of assuming that it's always django.contrib.auth.models.User.

[jesperhodge]

Maybe my misunderstanding.

openedx-core/docs/openedx_learning/decisions/0001-competency-criteria-location.rst

Line 48 in 845dc1f

- The learner status models in this feature would have a ForeignKey to settings.AUTH_USER_MODEL, which is a runtime/learner concern. If those models lived under the authoring app, then the authoring app would have to import and depend on the user model, forcing an authoring-only package to carry learner/runtime dependencies. This may create unwanted coupling.

[ormsbee]

So in theory, very little should write to the User model, but it's true that we have seen reports of concurrency issues that we don't understand very well at this point:

• feat!: rm user foreign key in db completion#443
• feat: bookmarks rm user foreign key db constraint openedx-platform#38815

On that note, if there is competency information that makes sense to attach in a 1:1 way per student, e.g. a CompetencyStudent model that would go 1:1 with User, it might make sense to make various competency models foreign key to that instead of directly to User.

In either case though, I think we can remove this decision from the ADR.

[ormsbee]

This is always a possibility for the future, but one we'd want to punt until we had verified that it was a problem worth tackling and testing for. StudentModule load is really high because it's used in many, many places, the payload can be enormous, and the lookup pattern happens via a very expensive/large composite index. My hope would be that even on sites that make heavy use of CBE, the load from it should be much lower.

[ormsbee]

I strongly suspect that this will end up changing in the future, but I'm okay with it as a starting position.

[ormsbee]

Again, I take the point that we shouldn't silently do this by default, but I expect people will have errors in their content settings, and this will need to be adjustable at some point. But also again, I'm not going to block this as a starting position.

Also, we've had mention of use cases in the past where mastery has a shelf-life, e.g. Continuing Medical Education (CME) credits.

[jesperhodge]

@ormsbee @mgwozdz-unicon @thelmick-unicon two open questions:

• Do we need the history table to keep records forever, or can we auto-truncate it after a certain period?
• If we only store advancing states in the history table, does that conflict with any edge cases? For example, if a learner grade gets corrected by an instructor, or a django admin corrects mistaken entries - does the correction need to be traceable in the history table?

[mgwozdz-unicon]

• Do we need the history table to keep records forever, or can we auto-truncate it after a certain period?

I think it would be safe to auto-truncate as long as the instance owner is able to control what the auto-truncation period is.

• If we only store advancing states in the history table, does that conflict with any edge cases? For example, if a learner grade gets corrected by an instructor, or a django admin corrects mistaken entries - does the correction need to be traceable in the history table?

I think we would want corrections and edits traceable in the history table as well.

[jesperhodge]

I think it would be safe to auto-truncate as long as the instance owner is able to control what the auto-truncation period is.

@mgwozdz-unicon I'll defer to you on this, since I don't really know what the pieces are that need to be read off of the history table. -> So I'll add auto-truncation, but what should the mechanism for instance owners be - an environment variable? And what should the default setting for this be?

I think we would want corrections and edits traceable in the history table as well.

@mgwozdz-unicon how about we make it so there are two routes for this?

1. we record only advances, no repeat attempts or evaluations, and we don't allow downwards corrections that result from criteria tree changes. This keeps the history table small
2. Only in the case where the assignment is directly changed by a course instructor or admin we record this change in the django history (even if it's downward).

[jesperhodge]

@ormsbee I addressed your feedback; only the two open questions discussed with Mary above are not handled yet.

[thelmick-unicon]

@jesperhodge

Do we need the history table to keep records forever, or can we auto-truncate it after a certain period?

I agree with Mary, this is going to be variable by institution, depending on when they will want to truncate the history. As for a default, my first thought is 2 years, but I think that's open to discussion. This might be a good question for Jason.

If we only store advancing states in the history table, does that conflict with any edge cases? For example, if a learner grade gets corrected by an instructor, or a django admin corrects mistaken entries - does the correction need to be traceable in the history table?

Again, I agree with Mary that manual changes should be captured and traceable in the history.

how about we make it so there are two routes for this?
we record only advances, no repeat attempts or evaluations, and we don't allow downwards corrections that result from criteria tree changes. This keeps the history table small
Only in the case where the assignment is directly changed by a course instructor or admin we record this change in the django history (even if it's downward).

Feedback we received from institutions was that they explicitly wanted to capture all repeat attempts and evaluations so both the instructor and student can see all attempts to evaluate if progress is being made or to review feedback from previous attempts.

[mgwozdz-unicon]

I think it would be safe to auto-truncate as long as the instance owner is able to control what the auto-truncation period is.

@mgwozdz-unicon I'll defer to you on this, since I don't really know what the pieces are that need to be read off of the history table. -> So I'll add auto-truncation, but what should the mechanism for instance owners be - an environment variable? And what should the default setting for this be?

Yes, I think an environment variable sounds good. Claude is recommending a Django setting as a precedent in the codebase. The 2 years Tammie suggested sounds good to me, though that's probably an implementation detail that can get worked out later. I put a question out to Jason about it.

I think we would want corrections and edits traceable in the history table as well.

@mgwozdz-unicon how about we make it so there are two routes for this?

1. we record only advances, no repeat attempts or evaluations, and we don't allow downwards corrections that result from criteria tree changes. This keeps the history table small
2. Only in the case where the assignment is directly changed by a course instructor or admin we record this change in the django history (even if it's downward).

I agree with Tammie that we definitely want to keep data for repeat attempts/evaluations. In general, we probably do want to allow recording downwards corrections whether the correction was authored by a person or by software since I'm pretty sure I recall that D2L Brightspace supports displaying a log of all grade changes to an assignment over time whether it was changed by a system or a specific person. That way we can support parity with what institutions already expect.

I think the main consolation here is that ADR 3 says "If a user edits competency criteria definitions or competency object/tag associations after related learner status exists, Studio must display an explicit warning that student statuses have already been set, and these changes will be applied going forward, so existing learner statuses will not be retroactively updated." So at least they won't get retroactively updated by the software in that case.