Why composable apps?

Make your algorithms more robust

As the robustness of POSIX based operating systems (think Linux, Mac OS, Unix) can attest, writing algorithms that stitch together multiple single purpose applications is a Very Good Thing™. This is most elegantly expressed as a part of the Unix design philosophy.

scinexus encourages this design pattern. We leverage the Python type annotation system to govern the compatibility (composability) of different applications. This enables in-process composition of your applications with validation of the consistency of the pipeline and the consistency of the data being run through it.

We can expand on this slightly for the problem of scientific computation by considering the critical benchmark of satisfying the conditions for reproducible computation, i.e. the obligation to track all of the properties affecting the execution of your algorithm. Examples of this are the operating system, the language version, the seed used for the random number generator, etc.

scinexus does this for you. For example, we intercept all arguments (including default values) passed to the construction of apps and record them so that the app state is logged. If you, the developer, also leverage the capabilities of the scitrack logging package (which scinexus has as a dependency), you can capture extra information such as versions of packages that your application depends on. We provide an example of using scitrack for these cases.

Improve the accessibility of your work for end users

Apps are ready-made functions that users can run on their data without needing technical expertise. They’re easy for non-programmers to use, and can be linked together into pipelines. This lets users process one or millions of records at once—without writing loops, conditionals, or other structural code.

When to use apps

Here are three examples of when to use apps.

To make a callable class

You want a class to be callable, optionally with its input and output types checked at runtime, without writing the boilerplate yourself.

To make a composite data transform

You want to chain several steps into a single transformation and reuse that combination across a code base. For example, converting an object to a JSON string and then compressing it involves two separate functions, plus possible configuration options for compression. By writing each step as an app, you can compose them into a single instance, assign it to a module-level variable. Importing and using that instance wherever it is needed ensures consistency in transformation behaviour.

To make a robust multi-step pipeline

You have a composite transformation that you want to apply to many data records. Written conventionally, this would require a loop that calls each step in turn, traps any exceptions, handles their consequences, and passes successful results on to the next step.

The apply_to() and as_completed() methods on composed apps handle this pattern for you. They iterate over the input records, run each one through the pipeline, and collect the results — so you do not need to write the loop, the exception handling, or the conditional checks between steps. And as an extra bonus, you can turn on parallelisation using a single argument.