-
-
Notifications
You must be signed in to change notification settings - Fork 973
Fix flaky WhereQueryClosureCaptureSpec: key AstPropertyResolveUtils cache by ClassNode identity #16034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 8.0.x
Are you sure you want to change the base?
Fix flaky WhereQueryClosureCaptureSpec: key AstPropertyResolveUtils cache by ClassNode identity #16034
Changes from 1 commit
620f651
078ac9d
c3a4d92
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,7 +20,9 @@ | |
| package org.grails.datastore.gorm.transform; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.Collections; | ||
| import java.util.HashMap; | ||
| import java.util.IdentityHashMap; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
|
|
@@ -47,7 +49,28 @@ | |
| * @since 6.1 | ||
| */ | ||
| public class AstPropertyResolveUtils { | ||
| protected static Map<String, Map<String, ClassNode>> cachedClassProperties = new HashMap<>(); | ||
|
|
||
| /** | ||
| * Cache of resolved properties per {@link ClassNode}. | ||
| * <p> | ||
| * Keyed by {@code ClassNode} identity rather than name. {@link ClassNode#equals(Object)} and | ||
| * {@link ClassNode#hashCode()} compare by {@link ClassNode#getText()} (essentially the class | ||
| * name), so a {@code Map} keyed by name - or even by {@code ClassNode} itself as the map key - | ||
| * treats any two distinct {@code ClassNode} instances that happen to share a name as the same | ||
| * cache entry. That collision is a real hazard for classes compiled without a package (common | ||
| * in tests and dynamically generated sources), and for the same source compiled more than once | ||
| * in separate {@code GroovyClassLoader}s: each compilation produces its own {@code ClassNode} | ||
| * instance that must never share cached property data with another compilation's instance of a | ||
| * same-named class. An {@link IdentityHashMap} avoids that collision entirely by comparing keys | ||
| * with {@code ==} instead of {@code equals()}. | ||
| * <p> | ||
| * Wrapped in {@link Collections#synchronizedMap(Map)} because AST transforms that populate and | ||
| * read this cache can run concurrently on multiple threads (e.g. parallel test execution within | ||
| * one JVM/fork); a plain, unsynchronized {@link HashMap} is not safe for concurrent structural | ||
| * modification and can corrupt its internal state under concurrent {@code put()} calls. | ||
| */ | ||
| protected static final Map<ClassNode, Map<String, ClassNode>> cachedClassProperties = | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two breaking changes to a If it's being broken anyway, take it the rest of the way: make it
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Went further than private + |
||
| Collections.synchronizedMap(new IdentityHashMap<>()); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keying this static map by Under the old plus Groovy already stores per-node state exactly this way ( private static final String PROPERTIES_CACHE_KEY = AstPropertyResolveUtils.class.getName() + ".properties";
private static Map<String, ClassNode> getPropertiesFromCache(ClassNode classNode) {
return classNode.getNodeMetaData(PROPERTIES_CACHE_KEY, cn -> computeProperties(classNode));
}
If a process-wide map has to stay for some reason, it needs weak identity keys and/or an explicit eviction point, plus a note documenting the expected lifecycle.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, and went with your suggested direction: the static map is gone entirely, and the cache is now stored as |
||
|
|
||
| /** | ||
| * Resolves the type of of the given property | ||
|
|
@@ -94,22 +117,25 @@ public static List<String> getPropertyNames(ClassNode classNode) { | |
| } | ||
|
|
||
| private static Map<String, ClassNode> getPropertiesFromCache(ClassNode classNode) { | ||
| String className = classNode.getName(); | ||
| Map<String, ClassNode> cachedProperties = cachedClassProperties.get(className); | ||
| Map<String, ClassNode> cachedProperties = cachedClassProperties.get(classNode); | ||
| if (cachedProperties == null) { | ||
| cachedProperties = new HashMap<>(); | ||
| Map<String, ClassNode> newProperties = new HashMap<>(); | ||
| boolean isDomainClass = AstUtils.isDomainClass(classNode); | ||
| if (isDomainClass) { | ||
| cachedProperties.put(GormProperties.IDENTITY, new ClassNode(Long.class)); | ||
| cachedProperties.put(GormProperties.VERSION, new ClassNode(Long.class)); | ||
| newProperties.put(GormProperties.IDENTITY, new ClassNode(Long.class)); | ||
| newProperties.put(GormProperties.VERSION, new ClassNode(Long.class)); | ||
| } | ||
| cachedClassProperties.put(className, cachedProperties); | ||
| ClassNode currentNode = classNode; | ||
| while (currentNode != null && !currentNode.equals(ClassHelper.OBJECT_TYPE)) { | ||
| populatePropertiesForClassNode(currentNode, cachedProperties, isDomainClass, !isDomainClass); | ||
| populatePropertiesForClassNode(currentNode, newProperties, isDomainClass, !isDomainClass); | ||
| currentNode = currentNode.getSuperClass(); | ||
| } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Identity keying makes the cache immune to name collisions, but I don't think it removes the nondeterminism that a flake like #16030 needs. That also weakens the root-cause story in the description: compiling the same source twice produces two Can you pin down which one you observed — e.g. the two colliding class names, or the flake reproducing in a loop pre-fix and not post-fix? The change is an improvement regardless, but if the resolution-state snapshot is the real driver then #16030 comes back and this gets recorded as already fixed.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wanted to actually test this rather than argue about it. I reverted to the pre-fix cache and ran the exact "same source compiled twice" scenario On the |
||
| } return cachedProperties; | ||
| // Publish only once fully populated so a concurrent reader can never observe a | ||
| // partially-populated entry for this ClassNode. | ||
| cachedProperties = newProperties; | ||
| cachedClassProperties.put(classNode, cachedProperties); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The reorder-and-publish-last fix is correct, and One remaining wrinkle: the check-then-act across the
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| } | ||
| return cachedProperties; | ||
| } | ||
|
|
||
| private static void populatePropertiesForClassNode(ClassNode classNode, Map<String, ClassNode> cachedProperties, boolean isDomainClass, boolean allowAbstract) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.grails.datastore.gorm.transform | ||
|
|
||
| import java.lang.reflect.Modifier | ||
|
|
||
| import org.codehaus.groovy.ast.ClassHelper | ||
| import org.codehaus.groovy.ast.ClassNode | ||
| import spock.lang.Specification | ||
|
|
||
| /** | ||
| * {@link AstPropertyResolveUtils} caches resolved property metadata in a static, process-wide | ||
| * map keyed by {@link ClassNode}. Two distinct compilations (e.g. the same source parsed in two | ||
| * different {@code GroovyClassLoader}s, as happens for dynamically-generated sources and in | ||
| * tests) produce distinct {@code ClassNode} instances that can legitimately share the exact same | ||
| * name - {@code ClassNode#equals(Object)} compares by name, so a naive name- or equals()-based | ||
| * cache key would conflate them, corrupting the resolved properties of one class with those of | ||
| * an unrelated class that happens to share its name. This spec proves the cache keys strictly by | ||
| * {@code ClassNode} identity, so same-named-but-distinct class nodes never contaminate each | ||
| * other's cached property data. | ||
| */ | ||
| class AstPropertyResolveUtilsSpec extends Specification { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good to see this class get its first coverage. As written, though, it only exercises hand-built non-domain Also missing: a test for the concurrency fix the PR claims. Several threads calling
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added both. Domain-class coverage now includes identity/version injection and Also added two concurrency tests: many threads each resolving their own distinct, identically-named |
||
|
|
||
| void "property lookups for two same-named ClassNodes in different packages do not corrupt each other"() { | ||
| given: 'two distinct ClassNodes with the same simple name declared in different packages' | ||
| ClassNode first = new ClassNode('org.example.one.Widget', Modifier.PUBLIC, ClassHelper.OBJECT_TYPE) | ||
| first.addProperty('color', Modifier.PUBLIC, ClassHelper.STRING_TYPE, null, null, null) | ||
|
|
||
| ClassNode second = new ClassNode('org.example.two.Widget', Modifier.PUBLIC, ClassHelper.OBJECT_TYPE) | ||
| second.addProperty('weight', Modifier.PUBLIC, ClassHelper.Integer_TYPE, null, null, null) | ||
|
Comment on lines
+56
to
+62
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch on the mechanics — you're right that this test's two ClassNodes use different fully-qualified names ( That guard is the next test below, This first test is intentionally a different, narrower check — that two distinct, differently-named classes never get confused with each other, which is a correctness property worth keeping on its own regardless of the collision bug. Leaving both as-is: this one for general non-contamination across genuinely different classes, the next one for the actual same-name collision regression. |
||
|
|
||
| when: 'the first class node is resolved, populating its cache entry' | ||
| List<String> firstProperties = AstPropertyResolveUtils.getPropertyNames(first) | ||
|
|
||
| then: 'only its own property is resolved' | ||
| firstProperties.contains('color') | ||
| !firstProperties.contains('weight') | ||
|
|
||
| when: 'the second, differently-packaged, same-simple-name class node is resolved' | ||
| List<String> secondProperties = AstPropertyResolveUtils.getPropertyNames(second) | ||
|
|
||
| then: 'its own property is resolved, not leaked from the first class node' | ||
| secondProperties.contains('weight') | ||
| !secondProperties.contains('color') | ||
|
|
||
| and: 'the first class node cache entry remains unaffected by resolving the second' | ||
| List<String> firstPropertiesAfter = AstPropertyResolveUtils.getPropertyNames(first) | ||
| firstPropertiesAfter.contains('color') | ||
| !firstPropertiesAfter.contains('weight') | ||
| } | ||
|
|
||
| void "property lookups for two distinct ClassNode instances with the exact same unqualified name do not corrupt each other"() { | ||
| given: 'two distinct ClassNode instances - as produced by two separate compilations - sharing an identical unqualified name' | ||
| ClassNode first = new ClassNode('Widget', Modifier.PUBLIC, ClassHelper.OBJECT_TYPE) | ||
| first.addProperty('color', Modifier.PUBLIC, ClassHelper.STRING_TYPE, null, null, null) | ||
|
|
||
| ClassNode second = new ClassNode('Widget', Modifier.PUBLIC, ClassHelper.OBJECT_TYPE) | ||
| second.addProperty('weight', Modifier.PUBLIC, ClassHelper.Integer_TYPE, null, null, null) | ||
|
|
||
| expect: 'the two ClassNode instances compare equal by name - the exact condition that would collide in a name-keyed or equals()-keyed cache' | ||
| first == second | ||
| first.hashCode() == second.hashCode() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These two assert Groovy's own
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done - trimmed to just |
||
| !first.is(second) | ||
|
|
||
| when: 'both class nodes are resolved' | ||
| List<String> firstProperties = AstPropertyResolveUtils.getPropertyNames(first) | ||
| List<String> secondProperties = AstPropertyResolveUtils.getPropertyNames(second) | ||
|
|
||
| then: 'each keeps its own, independently-resolved properties despite comparing equal' | ||
| firstProperties.contains('color') | ||
| !firstProperties.contains('weight') | ||
| secondProperties.contains('weight') | ||
| !secondProperties.contains('color') | ||
| } | ||
|
|
||
| void "getPropertyType resolves and caches the type of a declared property"() { | ||
| given: 'a class node with a declared property' | ||
| ClassNode classNode = new ClassNode('org.example.PropertyTypeWidget', Modifier.PUBLIC, ClassHelper.OBJECT_TYPE) | ||
| classNode.addProperty('label', Modifier.PUBLIC, ClassHelper.STRING_TYPE, null, null, null) | ||
|
|
||
| expect: 'the resolved property type matches the declared type, both on first (cache-populating) and second (cache-hit) lookup' | ||
| AstPropertyResolveUtils.getPropertyType(classNode, 'label') == ClassHelper.STRING_TYPE | ||
| AstPropertyResolveUtils.getPropertyType(classNode, 'label') == ClassHelper.STRING_TYPE | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The name says "and caches", but neither assertion can distinguish a cache hit from a miss — on a miss To actually pin the caching behaviour: resolve once, then add a second property to the
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Replaced it. The new test adds a second property to the |
||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This rationale is inaccurate in a way that will mislead the next reader.
maxParallelForks = configuredTestParallel(gradle/test-config.gradle:86) forks separate JVMs, and each JVM gets its own copy of astaticfield — parallel test forks can therefore never race on this map. JUnit's in-JVM parallel execution isn't enabled anywhere in the build either.The concurrency exposure that does exist is the compiler itself: multiple
compileGroovytasks running concurrently in a shared Gradle worker, and any embedded compilation driven from more than one thread. Worth rewording to that, otherwise the comment justifies the synchronization with a scenario that can't happen.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, fixed (078ac9d). The rationale no longer claims parallel test forks race on this - confirmed against
gradle/test-config.gradlethatmaxParallelForksreally does fork separate JVMs and this build never enables JUnit's in-JVM parallel execution, so that was never a real race. Rewrote the javadoc around the actual exposure instead: internedClassHelpersingletons (OBJECT_TYPE,STRING_TYPE, etc.) that any concurrently-running compilation in the same JVM could resolve to and touch through this cache - which is also why access is now synchronized per-node (c3a4d92) rather than resting on "only one thread ever touches a given node," which turned out not to be universally true.