Refactor communicator layer, integrate with SubDofHandler - #52
Refactor communicator layer, integrate with SubDofHandler#52termi-official wants to merge 2 commits into
Conversation
NODDofHandler now wraps a standard Ferrite.DofHandler on the local grid, gaining full SubDofHandler support: fields restricted to subdomains, including subdomains that do not intersect every partition. The legacy DofHandler fork and the ported ConstraintHandler/evaluation code are replaced by forwarding to the SubDofHandler-aware Ferrite machinery. Dof ownership and global numbering are negotiated per (entity, field) block in two rounds of neighborhood collectives on the interface graph, replacing the rank-serialized MPI.Send/Recv chains; the owned dof offset uses MPI.Exscan. The new InterfaceCommunicator (src/Communicator.jl) wraps the graph communicator and is reused by the PartitionedArrays and HYPRE extensions (the latter's ghost value sync is now a request/reply exchange). Entity dof queries (vertex_dofs etc.) are based on the entity dicts from Ferrite.__close! instead of cell layout arithmetic, making them correct across SubDofHandlers. CoverTopology/NODGrid now track shared edges in 2D, fixing the numbering of interior edge dofs (e.g. 2D quadratic interpolations) across ranks. Breaking: the legacy field query API (field_dims, integer field indices on the handler, entity dof queries on serial handlers) is gone; use the Ferrite SubDofHandler API instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Local node ids are now assigned in ascending global node id order, so node id comparisons agree between processes. This makes the canonical entity orientation (sortedge/sortface) identical on all ranks, meaning shared dof blocks with more than one dof per edge (e.g. cubic Lagrange) transfer correctly without extra orientation handling. Remove the corresponding restriction and test quadratic/cubic (scalar and vector) interpolations against the serial numbering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The PR looks good for review on my end. However, I would like to merge the GPU support and make a release first before merging this, as there might be some problems popping up in the dispatches (which might become ambiguous). As a follow up, before registering a v0.1, we must upstream the remaining debris in utils.jl to Ferrite. |
koehlerson
left a comment
koehlerson
left a comment
There was a problem hiding this comment.
Only remark I have is we may leave an issue/todo or whatever for unifying the code regarding EntityDofInfo with https://github.com/Ferrite-FEM/Ferrite.jl/blob/master/src/Dofs/DofHandler.jl#L110-L114
| ldh = getlocaldofhandler(dh) | ||
| _, vertexdicts, edgedicts, facedicts = Ferrite.__close!(ldh) | ||
| counts = _entity_dof_counts(ldh) | ||
| setfield!(dh, :entity_dofs, EntityDofInfo(vertexdicts, edgedicts, facedicts, counts)) |
There was a problem hiding this comment.
hm maybe at least a todo regarding the changes of the dofhandler on master storing optionally the dicts?
There was a problem hiding this comment.
I guess we should stall this until the AMR has a new Ferrite release.
There was a problem hiding this comment.
problem is that in the current master it checks explicitly for a NonConformingGrid and only stores in that case the dict
There was a problem hiding this comment.
3 participants
NODDofHandler now wraps a standard Ferrite.DofHandler on the local grid, gaining full SubDofHandler support: fields restricted to subdomains, including subdomains that do not intersect every partition. The legacy DofHandler fork and the ported ConstraintHandler/evaluation code are replaced by forwarding to the SubDofHandler-aware Ferrite machinery.
Dof ownership and global numbering are negotiated per (entity, field) block in two rounds of neighborhood collectives on the interface graph, replacing the rank-serialized MPI.Send/Recv chains; the owned dof offset uses MPI.Exscan. The new InterfaceCommunicator (src/Communicator.jl) wraps the graph communicator and is reused by the PartitionedArrays and HYPRE extensions (the latter's ghost value sync is now a request/reply exchange).
Entity dof queries (vertex_dofs etc.) are based on the entity dicts from Ferrite.__close! instead of cell layout arithmetic, making them correct across SubDofHandlers. CoverTopology/NODGrid now track shared edges in 2D, fixing the numbering of interior edge dofs (e.g. 2D quadratic interpolations) across ranks.
Breaking: the legacy field query API (field_dims, integer field indices on the handler, entity dof queries on serial handlers) is gone; use the Ferrite SubDofHandler API instead.