Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make nodes extensible and weak-referencable #82

Merged
merged 6 commits into from
Mar 18, 2020

Conversation

AstraLuma
Copy link
Contributor

Add standard dunder attributes to (almost) all instances of __slots__

Using the test suite as a benchmark, this change produced no noticeable slow down.

Closes #81

@AstraLuma AstraLuma requested a review from Cito as a code owner March 17, 2020 17:25
@AstraLuma
Copy link
Contributor Author

I did not touch src/graphql/validation/rules/overlapping_fields_can_be_merged.py

@Cito Cito requested a review from syrusakbary March 17, 2020 20:59
@Cito
Copy link
Member

Cito commented Mar 17, 2020

I did not touch src/graphql/validation/rules/overlapping_fields_can_be_merged.py

Good, that PairSet class is only for internal usage anyway.

@Cito
Copy link
Member

Cito commented Mar 17, 2020

Thanks for sending a PR. Not quite sure if we really should add these to all classes, like Location (attached to every node) and make Token (mainly for internal use in the lexer) extendible.
Would like to get @syrusakbary's opinion regarding this PR.

@Cito
Copy link
Member

Cito commented Mar 17, 2020

In order to get a better feeling for the memory penalty, I ran memory_profiler while parsing the big_schema_sdl now, and found that for each of the 3 classes in the ast module, it uses 4% more memory, 12% more in total. I think this is pretty significant already. So I suggest relaxing only the Node class where it makes most sense. What do you think?

@AstraLuma
Copy link
Contributor Author

Sure, I can do that. My use case is primarily about Node anyway.

@AstraLuma
Copy link
Contributor Author

How do I do the memory profiling?

@Cito
Copy link
Member

Cito commented Mar 17, 2020

What I did was loading the github_schema, then running parse() on it, and observed the memory increase in that line using the memory_profiler. You can also try with other functions. But of course there must be many nodes in the game.

@AstraLuma
Copy link
Contributor Author

Ok, some variations. Using this script, each is an average of 3 sequential runs

  • master: 42.2MB / 17.8MB
  • This PR, as-is: 44.6MB / 19.9MB
  • Variation 1: 43.1MB / 18.5MB
  • Variation 2: 43.9 MB / 19.3MB
  • Variation 3: 43.9MB / 19.3MB
  • Variation 4: 44.3MB / 19.8MB

Ok, variations:

  • Variation 1: Dropped changes to Location and Token
  • Variation 2: Dropped changes to Token
  • Variation 3: Dropped __dict__ on Location and Token
  • Variation 4: Dropped __dict__ on Token

I was surprised by the memory impact, even between master and variation 3 (presence or removal of __weakref__). All that memory is just space from the extra slots--__weakref__ is initialized to None, which is a singleton. (Under the hood in CPython, it's an actual value, PyNone; PyObject* is not normally NULL.)

I guess I'll go with variation 1, then.

@Cito
Copy link
Member

Cito commented Mar 18, 2020

Thanks a lot for the detailed verification. I agree variation 1 is the best compromize.

This should decrease the memory impact of this PR from +11% to +4%.
@AstraLuma
Copy link
Contributor Author

Ok, I think that's all squared away.

@Cito Cito removed the request for review from syrusakbary March 18, 2020 11:31
@Cito Cito changed the title Fix slots Make nodes extensible and weak-referencable Mar 18, 2020
@Cito Cito merged commit faa80af into graphql-python:master Mar 18, 2020
@AstraLuma AstraLuma deleted the fix-slots branch March 18, 2020 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Weakref unsupported in AST
2 participants