How floro works (part 1)
(skip to part 2)
While git repositories are composed of files, floro repositories are composed of plugins and their corresponding state stores. Plugins are small UI applications directly managed by floro. If you are familiar with redux, you can consider a floro repository as a global redux state store made up of a subset of plugin state stores.
Each plugin defines a schema, which outlines the type of content persisted by that plugin. Plugins can read and write to an external upstream plugin's store when the plugin declares the upstream plugin as a dependency. In this sense, plugins function both like applications and programming dependencies. Plugins can also establish relationships between their internal store structures and the external structures of upstream plugins. Plugin developers can specify cascading behaviors in a plugin's schema, making plugins especially valuable for centralizing related content.
Floro utilizes something called dependent types to generate well-typed content. Ultimately, a plugin's state is designed to be used as input to a program called a generator. Generators transform the state of a plugin (usually into a type of static map). The generated code can then be utilized by an application. A generator's input is agnostic to language and environment, while it's output can target a specific runtime. In the example below, it is demonstrated how a Typescript output might be generated from the state of a simple localization plugin. It's important to note, there's nothing fundamental about floro that necessitates the generated code to be in Typescript or Javascript. A plugin may have multiple generators, and generators can take multiple plugin states as input. Generators can produce any type of output.
An instance of a typical Floro pipeline:
Plugin Schema -> Plugin State -> Generated Code
The core idea of floro is that by exposing the plugin schema to the system, floro is able to exploit the knowledge of a schema to diff and merge any document that has a schema conforming to the floro schema language.
Floro does not really ever see the plugin state as a tree state. Instead it treats plugin state as a partially ordered list of key value pairs. If you have any experience working on merge technology, you likely know that once you can describe a data structure as an ordered list or ordered set, you can merge it!
Essentially, on every state change, floro serializes a repository's tree state (plugin state) into an intermediary KV state. Floro can convert in both directions between the tree state and KV state of a given plugin. Below is an illustration of the two state representations for our example above.
Plugin Schema: KV State <-> Tree State
Disclaimer: This is not exactly the syntax floro uses and is a simplification of the actual floro syntax (it has some small inaccuracies).
So what is this bizarre path key syntax?
It's basically a path to the struct (the value of the key-value pair) inside the tree-state representation.
The $ syntax is used to select from a specific plugin state store. $(LocalizedKeys) is the root for the plugin, so it gets wrapped by $(<plugin-name>).
Next we have this weird ".Key<Welcome Hero>" bit. What it basically says is look in the list (LocalizedKeys) and search for the first object with a property called "Key" that has a value equal to "Welcome Hero". If you look in the plugin schema, you might notice that the Key property of the LocalizedKeys set has the property value pair "isKey": true. You can think of this property as a primary key for that structure.
At this point in the iteration, we've successfully traversed to the "Welcome Hero" struct. Next we select the ".Translations" part of the path and have another list, this time with two structures at the current nesting level. Again, we see the Key<Value> syntax, but this time with ".Locale<English>". So we search the list for the first element with a Locale equal to English.
And that's it!
All the Intermediate state representation is, is a list of key-value pairs of the form:
{
key: identity path,
value: object
}
By applying our list of ordered key pairs, we can reconstruct a state tree. Conversely, we can recreate any ordered key pair list, from any state tree.
When things change...
We will get to the version control side of things soon. We promise! The thing we're really interested in figuring out at this point is if there is anything we can universally guarantee about the code we generate from a generator. If we look back at our first example we can observe something kind of neat now. We can actually determine if an update to the plugin state, is type-compatible with our current content, by looking at the keys of our KV List alone.
To show this, we have to consider three types of changes.
- Addition - We will add a new translation phrase to our LocalizedKeys "Logout", and provide English and Chinese Translations for the new phrase
- Update Value - We will change the content value of our Localized Key for the phrase "Welcome Hero" from the phrase "Welcome to our website" to "Welcome to our app".
- Removal - We will remove the Chinese translation for our "Welcome Hero" translation
So now our question is actually pretty simple. We want to know if we can safely update version 1 to each new version N (version 2, version 3, and version 4). To do this we are going to ask a simple question, "are the keys of version 1 all present in the subsequent version's key list?"
We're going to use two tiny pieces of math (set) notation here, "⊆" and"⊄".
If you've never seen these symbols before or need a quick refresher, they're quite simple.
A ⊆ B, says A is a subset of B.
For example, if A = {1, 2, 3} and B = {1, 2, 3, 4}, we would say that A is a subset of B, since 1, 2, and 3 are all present elements in B.
A ⊄ B, says A is NOT a subset of B.
For example, if A = {1, 2, 3} and B = {1, 3, 4}, we would say that A is NOT a subset of B, since the element 2 is not present in the elements of B.
So how does this connect back to generating code and type safety? Well if we know the plugin schema, we are able to perform type compatibility checks between plugin states, simply by asking variations of the question stated above.
This is a really powerful trick, since what it allows us to do is make a certain assumption about our content. If we managed to deploy a version of our application that was built with "version 1" of our floro state, we could safely deploy updated versions of our app to versions 2 or versions 3 with full confidence that we are not causing any bugs without having to recompile our entire application. If however, we were to swap out our content from version 1, to version 4, we would likely crash if a user ever visited a page calling the key "Welcome Hero" for the "Chinese" version of the app.
By performing this simple set check, we've essentially done the work of a (relatively unoptimized) compiler on our content. Obviously, depending on the application and type of content, static analysis is still often necessary, however, we can use static analysis in conjunction with set checks to live update things that would otherwise seems very high risk to update without recompiling.
It's all related 🍄
Earlier we mentioned that floro supports relationships between the substructures of a plugin's state. How it accomplishes this is through references. The keys of the key-value pairs are the actual pointer values in references. While this syntax could be used to point to any structure in a tree-state, it only makes sense to hold references to the elements of a set.
To further explain, we're going to build upon our prior example of our LocalizedKeys plugin. First we're going to refactor our schema to include a second set called "Languages". In this example, we're syntactically treating Languages as a separate plugin. However, it is perfectly fine to have multiple root level sets in the same plugin.
If you're overwhelmed by this change, don't be. It's actually not nearly as complicated as it may look at first glance. If we look in our Translations data structure, you'll notice we abandoned our "Locale" property, which previously served as our primary key. We updated our data structure to now use a reference as a primary key.
We also set a cascading option "onDelete" to "delete", on our plugin schema for the LanguageReference property. The alternative cascading delete option value is "nullify". You probably can guess the difference at this point but if the "onDelete" value of a reference type is "delete", the set element containing that reference will be deleted, when the reference is deleted. For example, if we were to remove the Languages' element with the Code equal to "ZH" from the Languages list, our Translation element with the reference $(Languages).Code<ZH>, would also be removed. However, if the "onDelete" property were set to "nullify", our Translation element would not be removed, we would be left with a null pointer though (ie. the value of LanguageReference would be set to equal to null). It is important to note that nullifying our reference is not possible in this example, since the LanguageReference property is the primary key of the Translations set.
There's actually one more cascading behavior not commonly recognized in the world of relational databases that floro heavily utilizes, which is expanding a set by the boundaries of another set.
Imagine we wanted to add a new language (Spanish) to our plugin. If this were a relational database, we might add a new column "Spanish" to our Languages' table or make use of a join table between localizations and translations. Floro has a slightly different way of handling this relationship. We specify a property in our schema, that our set is a bounded set, by writing "bounded": true. This only works, when the primary key of our set is a reference to another set it can be bounded by. At this time floro cannot bind sets to other nested sets, only root-level sets.
Once we add Spanish to our language list, floro will automatically cascade all of the LocalizedKeys in the plugins state to include a reference to Spanish as well.
Bounded Cascading:
The above image illustrates how cascading enforces bounded sets. Since Translations are bounded by Languages (due to the primary key of Translations being a reference to Languages), we can guarantee that for every LocalizedKeys element, for each of its Translations element there will always be an element with a reference to each Language element. This becomes particularly important when we get into merging states, which necessitates cascading states.
Conceptually, we've now created for ourselves, what is essentially a spreadsheet.
If you're still with us, read on to see how diffing and merging work (part 2).