Follow

So, I'm working on this parallel workflow execution framework in using a implementation (datalevin) as temporary store.

Basically you define a job with some tasks, the relations between tasks and the framework took care to execute them in the right order (topological sorting and so on).

Parallelism and concurrency are achieved with clojure.core.async.

Using datalog as query language gives some help managing jobs and tasks relations: it's almost like having a graph db.

Of course we are going to release it when ready.

I can write about it a little bit more if anyone is interested.

@mdallastella Yes, please. This sounds to me a little bit like which I had looked into when I had the need to find a nice way to transform documents. In the end, I just implemented a much simpler thing based on core.async/pipeline that was really easy to come up with.

@schaueho yes, the idea was to build something like Onyx, we really like the way you describe jobs and tasks, but we also wanted something "smaller", with almost no external dependencies.

We are not on Onyx level at all right now, but we'll work on it. 😊

@mdallastella Cool. Looking forward to see something more concrete. 👏​

Sign in to participate in the conversation
Functional Café

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!