Dynamically typed queries

[Ralith]

hecs could be extended with support for executing queries constructed at runtime rather than specified statically in types. This could be useful for e.g. interactively user-specified queries or for running queries in embedded scripting environments.

An implementation could look something like:

impl World {
    fn query_dynamic(&self, q: DynamicQuery) -> DynamicQueryBorrow<'_>;
}

pub enum DynamicQuery {
    Get(TypeId),
    GetMut(TypeId),
    And(Vec<DynamicQuery>),
}

where DynamicQueryBorrow<'_> contains an iterator which yields DynamicQueryItem which has

impl DynamicQueryItem<'_> {
    fn get<T: Component>(&self) -> Option<&T>;
    fn get_mut<T: Component>(&self) -> Option<&mut T>;
}

or perhaps

impl DynamicQueryItem<'_> {
    fn get(&self, ty: TypeId) -> Option<&dyn Any>;
    fn get_mut(&self, ty: TypeId) -> Option<&mut dyn Any>;
}

This is complex enough that I'm not likely to work on it unless there's significant demand, so please leave a comment describing your use case if you're interested!

[Grinshpon]

This is definitely interesting. I don't have an ultra-specific use case, but I'm drawing out a framework I'd like to make that utilizes the advantages of an ECS architecture but still provides some form of scripting component for fast iteration and smoother workflow.

Plus it'd be nice to get around Rust's compile times by writing non-performance-critical systems in a script.

[Ralith]

Concrete use cases are particularly important because they can help inform the exact shape the API could take to solve a specific problem.

[Grinshpon]

I realize that. When I can write out a concrete case I'll come back to this issue to describe it.

[xacrimon]

I'm quite interested in this. I'd like to experiment with hooking hecs into the scripting language in my engine.

[Ralith]

Bevy is exploring a similar effort, which may at least provide inspiration. Of particular interest is the introduction of dynamic components, treated as plain fixed-length data associated with a custom ID.

A key question for proceeding with API design here is what's required for passing results to a scripting layer easily and efficiently. Examples of the patterns needed to provide low-overhead and ergonomic access to Rust data from a script would be valuable.

[Ralith]

Another compelling idea from bevy: define a space of ComponentIds independent of TypeIds, map from TypeIds to ComponentIds for conventional use, and allow explicit allocation of ComponentIds for dynamic cases.

These could be feature-gated to allow users who only need static components to avoid the indirection. Dynamic queries could still be available regardless.

[sdleffler]

Was chewing on this lately and was able to come up with a solution which essentially works by trait-object-izing Fetch, and making DynamicQuery a type alias of Box<dyn for<'a> ErasedFetch<'a>>, and using DynamicFetch and DynamicState types to erase types and mirror the implementation in query. The downside of this is that DynamicItem ends up being created on every entity satisfying the query, and since the structure of DynamicItem in my experiment is essentially a syntax tree that almost exactly mirrors the structure of DynamicQuery, in the case that multiple items are returned as part of the DynamicItem, a Vec is allocated. This could be optimized as a tuple I guess, by assuming that query results will be under some static count? Or by parameterizing DynamicItem/DynamicQuery on some const usize and using SmallVec to avoid heap allocations as much as possible. Alternatively it wouldn't be hard at all to allocate DynamicItems (at least in the scheme I'm using now) in an arena allocator... Probably overthinking it at this point; my brain's a bit fried after spending most of the day on it, lol. Should be a working PoC though.

[All8Up]

This is actually the reason I was reading through the issues in the first place. I'm not sure what the state of this is but it is a requirement for my work and the potential of switching the current C++ component storage/query system to hecs. (I'm tired of hunting bugs in C++ and I have rewritten a bunch of my code in Rust, so....) Basically the three key bits I'm looking at as being blockers for my intentions:

1. I will be required to expose the basic component/system creation to a C API so the legacy code does not all have to be rewritten. I end up wrapping this at a higher level in my proof of concept work since I have to tack on a set of uniform components (aka: shared by all entities), i.e. my current systems are executed as a Fn(U: (tuple of uniform), V: (hecs tuple of varying)) and while I solved the uniforms and it all works with compile time generation, I just started looking at how to dynamically build the varying components, hence this post.

2. One of the core system types is actually generated from descriptor files and hot reloaded on edits. It does not know the components at compile time and only has them when the descriptors are loaded. I've currently gotten this to the point of generating the vector of TypeInfo's using from_parts (pretty sure that's the purpose?), but then I ran into this, hence the post.

3. The core system mentioned above is a behavior tree, it has two differences compared to most: it is multicore and it generates multiple systems to be executed in series. It needs the multiple system generation to break up the component access into thread legal behaviors such: as find all nearby entities, read their positions/velocities ... then compute the local entity's new position/velocity. Obviously in a threaded scenario the reads and the writes have to be separated into different systems with a barrier between them to prevent data races. So, not only do I not have the components up front, I don't even know how many systems I will end up inserting into the processing pipeline.

Anyway, this enhancement seems like the answer, correct?

[Ralith]

Some of the interfaces discussed here could be applied to those problems, yeah. See also the draft in #202, which has been successfully applied to Lua, which should be a superset of the challenges presented by a C binding.

You describe a dynamically varying C API--are you working with a shared library plugin system?

[All8Up]

Some of the interfaces discussed here could be applied to those problems, yeah. See also the draft in #202, which has been successfully applied to Lua, which should be a superset of the challenges presented by a C binding.

I'll take a look at that, though one of the key legacy items will be problematic if the performance degradation is significant. I.e. the render instance components tied to the C++ side renderer. I haven't convinced those guys to become Rusty yet, it's on the long term evil corruption todo list though.

You describe a dynamically varying C API--are you working with a shared library plugin system?

No, I won't be using dll's/so's/dylib's etc for this. They are way too problematic in multiple ways., especially with Windows where half the time they won't tear down at all due to PDB's being non unloadable, file locking issues and tons of other problems. When I said "dynamic" I was referring to the hot reload of the behavior tree's because with those the things changing are programically generated. I have a number of similar items which will have the same issues, they don't have a fixed system signature because they are generated from data. The more complicated example would be the LLVM Orc JIT I use during development, it compiles the "plugin" straight to memory and I dig out the handles to the C API functions that will need to have access to the components in hecs based on what I find in the signature.

Typically speaking though, with the current system there is very little performance loss when doing this because the bindings are generated just once and reused until such time as something reloads. I'm hoping to get similar behavior here though it's not absolutely required since after things start working you flip a switch and the code side items just get compiled into the final executable and the hot reload is removed, leaving a load/bind only once in it's place.

[Ralith]

Cool; please do leave feedback in the PR regarding its usefulness to you!

[Jerrody]

It's a very good idea to use Python for the caller side. Where via pyo3 crate Python's code calls Rust code. It's not so hard to implement and it's a very effective way to do stuff.

And generally, using Python for the scripting language for Engines it's really cool idea.