Formbot

Template

Our forms are represented internally as recursive JSON objects which we call templates. The EditForm page reads a template and transforms it into a fill-out-able form. The form builder, then, is just a very visual, glorified JSON builder; its only job is to create that JSON object.

Here's the shape of a template:

type Template = Gadget

interface Gadget {
  /////////
  // Keys that only exist on Layout Gadgets:
  /////////

  // A list of other gadgets to be rendered in your container. These will be passed to you, pre-built, in your View and Edit functions (see "Gadget Definition")
  children?: [Gadget]

  /////////
  // Keys that only exist on Data Gadgets:
  /////////

  // Data is stored on a document keyed by this. The end user cannot change this
  formKey?: string
  // If true, the forms service will scrub this data before returning it to the client
  secret?: boolean
  // Whether or not this field is required to be filled out for submission
  required?: boolean
  // This is a list of validations to apply to this gadget. For example, giving a text field a regex check or min/max for a number. Details about these validations can be found in https://github.com/kualibuild/builder-ui/blob/master/app/src/formbot/validations/index.jsx
  validations?: {
    [key]: {
      enabled: boolean
      value: V
    }
  }
  // If enabled, this name will be used to identify this gadget in the document list, gadget dropdowns, and anywhere else in the UI
  customName?: {
    enabled: boolean
    value?: string
  }
  // If enabled AND if you hit the graphQL api with the right query, the document(s) you receieve back from our API will have this gadget's data stored under this key instead of the formKey
  customFormKey?: {
    enabled: boolean
    value?: string
  }

  /////////
  // Keys that can exist on all Gadgets:
  /////////

  // Every gadget has a unique, unchangeable id
  id: string
  // This is a string matching any registered gadget type. i.e. "Section" or "Text"
  type: string
  // The label is optional for layout gadgets, required for data gadgets
  label?: string
  // Gives the user more clarification about why this field is important
  description?: {
    enabled: boolean
    displayAsPopover?: boolean
    value?: string
  }
  // All the custom configuration specific to a gadget is stored here. You can find more details on this in "Gadget Definition"
  details?: GadgetDetails
  // Turn this on if you'd like this gadget to be shown/hidden based on other data in the form
  conditionalVisibility?: {
    enabled: boolean
    // This is the set of rules we'll evaluate to determine if your gadget should show or not
    value?: {
      // choose whether all of the rules must pass to show your gadget, or if only one needs to pass
      type: 'any' | 'all'
      parts: [
        {
          // the part of the form you depend on. Note this key is prefixed with `data.`, `meta.`, or `integration.` which means you can depend on any part of the document; not just the things the user filled out
          formKey: string
          // The data here follows a format defined by the gadget identified by formKey on the line above. In other words, unlike GadgetDetails and GadgetValidations: this GadgetPdData probably won't be the one defined by this gadget type. For example, let's say this is a Section that should show up if some other Text gadget contains the value "test". While most everything else in this json relates to Section, this GadgetPdData would be the one defined in Text
          data: GadgetPdData
        }
      ]
    }
  }
}

You'll notice that a lot of those keys are objects with enabled and value. This is to support the configuration UI. A config panel shows up on the right of your screen when you drag a gadget onto the form in the form builder. Many of the options in that panel are toggled off by default. Toggling them on sets enabled to true, allowing you to fill out value.

You'll also notice I didn't define what GadgetDetails, GadgetValidations, or GadgetPdData look like. Each of them is an object but their shape is different based on the Gadget type. I'll make note of how those are created below in "Gadget Definition".

Document

The template defines how a form should be displayed and which controls should show up. When a user fills out that form they create a Document. While the JSON structure representing a template is very recursive, documents are relatively flat.

interface Document {
  // Yup; everything has an id
  id: string;
  // Identifies which form this document belongs to
  formContainerId: string;
  // Identifies which version of the form this document belongs to
  formId: string;
  // We don't do hard deletes. If this is true the document won't show up in lists
  deleted: boolean;
  // Whether or not the document has been submitted. We allow saving an invalid or incomplete document while in a draft state. Once it makes it to published we expect the document to be 100% valid. Note: even in a draft state we do not allow saving a document wherein one of its gadgets has failed its validateShape check. See below for more info on validateShape
  status: 'draft' | 'published';
  // This is autogenerated on the server. It's a collection of all the text in data, meta, and integration and is used for full text search
  keywords: [string];
  // All of the data entered by the user. It's an object keyed by formKey with values defined by each Gadget Definition. This object must strictly match the document's template. We don't allow extra keys and we enforce that GadgetValue matches the expected shape
  data: {
    [formKey]: GadgetValue;
  }
  // All of the data that our systems add to the document. If you want data from here to be usable in the form you must tie your data to a gadget type registered in formbot
  meta: {
    // The user that created the document. Will be empty for anonymous forms
    createdBy?: User;
    // The user that hit the submitted the document. Will be empty for anonymous forms. Today these users are always the same
    submittedBy?: User;
    // When the doc was created...
    createdAt: number;
    // ...aaaand when it was submitted
    submittedAt: number;
    // The user that last updated the document.
    lastModifiedBy: User;
    // ...and when it was last updated by that user
    lastModifiedAt: number;
    //
    appPublishVersion: number;
    // A human-readable number identifying this doc. We start with 0001 and increment from there. This counter is not shared across the system; each App starts at 0001
    serialNumber: string;
    // All the data needed to render our workflow tracker screen. It's a pretty expensive object to compute so we cache it here after workflow finishes running instead of creating it at query time
    simulation: object;
    // Shorter summary of where the document is at in workflow. Generated at the same time as simulation but used primarily in the document list
    workflowStatus: 'Draft' | 'In Progress' | 'Error' | 'Withdrawn' | 'Rejected' | 'Complete';
    // How long this document has been in workflow since being submitted. This number is viewable/sortable in the document list and is rendered in TimeAgo format
    workflowRuntime: number;
    // How long this document has been in workflow since the document was first created. This number is viewable/sortable in the document list and is rendered in TimeAgo format
    workflowTotalRuntime: number;
    // Timestamp of when the workflow for this document completed.
    workflowCompletedAt: date;
    // ???
    currentWorkflowSteps: [???];
  }
  // System integrations that run in workflow have the ability to return and store data back on the form. This is where that data gets stored
  integration: {
    [wfStepId.integrationNodeId]: GadgetValue;
  }
}

Again, you'll notice that I didn't define what GadgetValue looks like. That is also defined differently for every gadget type. You'll see it below.

Gadget Definition

The real power of formbot comes from Gadgets! As you'll see in the definition below, they do a lot. These definitions are typically stored in a manifest.jsx file.

// The following types are all treated as opaque types outside of this gadgets definition. You should feel comfortable updating these types as long as you update every part of the gadget definition that creates, updates, or consume them. If this gadget is being used in production, updating these types will require a data migration

// This represents the shape of data that your gadget stores and manipulates
type GadgetValue = any;
// This is the configuration you expose to the end user and use within your gadget
type GadgetDetails = any;
// If your gadget supports being used in conditional visibility, then this type is the data needed to perform your visibility check
type GadgetPdData = any;
// This data represents your filter for gadgets that support filtering on the document list
type FilterValue = any;

interface GadgetDefinition {
  // Whether this is a layout gadget or a data gadget
  layout?: boolean;
  // Given a config and a value, return a human-readable representation of your value. This is used in view-mode forms, on the document list, and in some cases on the edit-mode form. This component, as well as the Edit component below, are both passed a slew of other props. You can see what those are here: https://github.com/kualibuild/builder-ui/blob/master/app/src/formbot/engine/formbot/gadget-renderer.jsx
  View: ({ details: GadgetDetails, value: GadgetValue, ...others }) => ReactElement;
  // Given a config and a value, render an edit mode for your value. This is used in edit-mode forms and in Static Formbot. The argument you pass to onChange is saved and becomes your current value
  Edit?: ({ details: GadgetDetails, value: GadgetValue, onChange: Function, ...others }) => ReactElement;
  // This is how you ask your use for any required configuration for your gadget to work. This will be rendered at the top of your gadget's config panel. value is your full config object. You can update it by passing a full config object as the argument to onChange. Gadgets are part of Static Formbot and are document in the next section
  RequiredConfig?: ({ value: GadgetDetails, onChange: Function, Gadgets: SFBGadget, ...others }) => ReactElement;
  // Same as above but this config is considered optional. You must render each chunk of your optional configs in a ConfigBox to fit in with the current UI. Note that both RequiredConfig and OptionalConfig are passed the full config object, not just the pieces they care about. If you save just the pieces you care about for OptionalConfig, you could be erasing the data needed in RequiredConfig. Always save the full config object
  OptionalConfig?: ({ value: GadgetDetails, onChange: Function, Gadgets: SFBGadgets, ...others }) => ReactElement;
  // An Assembler takes all the pieces of a form control (label, description, control, errors, etc) and assembles them together. By default your gadget will use the Basic Assembler. If you want to override that, put something here
  Assembler?: ReactComponent;
  // This is just a different way to choose your Assembler. Use this instead if you just want to use one of the built-in assemblers
  getAssembler?: (assemblers: object, gridded: boolean) => ReactComponent;
  // This is only used on the Form Config page. It describes how your gadget looks in the gadget-picker sidebar
  meta: {
    // Whether or not you even want your gadget to show up in the sidebar
    hidden?: boolean;
    // What category should your gadget be displayed under
    category: 'Layout' | 'Basic' | 'Special' | 'Smart';
    // An svg identifying your gadget
    Icon: ReactElement;
    // The human-readable label for your gadget
    label: string;
    // If set, a help icon will be displayed next to your gadget. On hover, this text will be shown
    help?: string;
    // If set, a deprecation icon will be displayed next to your gadget. On hover, this text will be shown
    deprecated?: string;
    // If provided, this is invoked when your gadget is first placed on the form. The results will be splatted into your template. You can use it to set a default label, required, a partial config, etc
    initialTemplate?: () => object;
  }
  // If you'd like to sort your data on a nested key you can specify that here
  sortSuffix?: string;
  // If you support filtering on the document list, this is where you'll set that all up
  filters?: {
    // Here's where you define what your filter ui looks like. value is your current filter state and onChange is how you update it. You also get your complete template in case you need it.
    UI: ({ gadget: Gadget, value: FilterValue, onChange: Function }) => ReactElement;
    // We display a pill to the user so they know a filter is applied. The default one is pretty good but you can override it if you need to.
    Pill?: ({ label: string, filter: FilterValue, gadget: Gadget }) => ReactElement;
    // Because we give users the option to persist these filters to the server, we need the gadget to provide a way to serialize/deserialize their data. These two functions are that way. This one takes in your filter state and returns a serialized version...
    toGraphQL: (filter: FilterValue) => T;
    // ...and this one reverses that operation
    fromGraphQL: (raw: T) => FilterValue;
  };
  // If you want your gadget to be available as a conditional visibility input, this is where you implement that
  progressiveDisclosure?: {
    // When someone turns on conditional visibility in the config panel of a different gadget, and they pick your gadget from the provided list, this Component will then be rendered. It's your job to gather all the info you need to make conditional visibility work. value is your current state, onChange will update that
    component: ({ gadgets: ???, value: GadgetPdData, onChange: Function, details: GadgetDetails }) => ReactElement;
    // It's possible that a user didn't finish filling out the needed data for you to make a good call on conditional visibility. This function is where you can determine that. If the config is incomplete, return false
    configIsValid: (config: GadgetPdData) => boolean;
    // If conditional visibility is turned on AND the data you need is all there, then we'll call this function to find out if we should or should not show the gadget in question. You'll get the value that was filled out in your gadget as well as the data filled out in the conditional visibility config. Your job is to return a true (yes, please show this gadget) or false (nope, gotta hide it). You are also passed the entire document as a 3rd arg but that one is rarely needed
    check: (value: GadgetValue, config: GadgetPdData, document: Document) => boolean;
  };
  // When a document is first created, we will loop over all the gadgets in its form and use this key to generate a blank document. We also use this when doing required checks: if the stored value exactly matches this defaultValue it is considered unfilled and therefore the required check will fail
  defaultValue: GadgetValue;
  // deprecated. I don't believe this is actually used anywhere so we should get rid of it
  sampleValue: GadgetValue;
  // This is a function that takes in our custom validation library (formbot/engine/validate) and generates code to validate whether this gadget conforms to the GadgetValue type or not. It's used mostly on the server to validate that we're saving good documents. It's also used when displaying subFields to prevent those gadgets from crashing if invalid data is provided
  validateShape: Function;
  // This defines the set of validations that a user can choose to apply to this gadget.
  validationOptions?: [{
    key: string;
    label: string;
    UI: ({ Gadgets: SFBGadgets, details: GadgetDetails }) => ReactElement;
    evaluate: (value: GadgetValue, inputs: V, gadgetDetails: GadgetDetails) => [string] | null;
    description?: string | ReactElement;
  }];
  // This is exactly the same as validations on the template. When set here, however, the validations will always apply and are not user configurable. I believe this is only used in the email gadget today to apply an email regex check on every email gadget
  validations?: {
    [key]: {
      enabled: boolean;
      value: V;
    }
  };
  // This allows gadgets to expose other gadgets. Think about a UserTypeahead. It stores a user object which also has a schoolId and an email. We want users to be able to use those extra fields in workflow, conditional visibility, the document list, etc. By setting up subFields, users can now use a Text gadget for a UserTypeahead's schoolId or an Email gadget for that UserTypeahead's email. You can see what fields are exposed in the config panel under the `Add linked auto-filled gadgets` section
  subFields?: ({ id: string, formKey: string, label: string, details: GadgetDetails }) => [{
    // The attributes passed in to this function are those of the parent gadget. The ones you're returning in this array should be those of any sub-gadget you want to expose from this one
    id: string;
    type: string;
    formKey: string;
    label: string;
    details: GadgetDetails;
  }];
}

Some gadgets are only used in the document list. Only View is required for those ones though they could also benefit from filters. For gadgets that appear on the form View, Edit, defaultValue, and meta are all required. Most keys are optional and adding them will opt you in to more functionality.

Static Formbot

To further benefit from the effort expended to make gadgets, we've created a component called StaticFormbot. This component allows you to use gadgets to build forms in jsx code rather than building the form from a json template. Here's an example of what that looks like:

const [state, setState] = React.useState({})

<StaticFormbot value={state} onChange={setState}>
  {Gadgets => (
    <Gadgets.Section
      label='Config'
      description='This is where you configure the thing'
    >
      <Gadgets.Text label="Placeholder" configKey='placeholder' />
      <Gadgets.Number label="Amount" configKey='amount' />
    </Gadgets.Section>
  )}
</StaticFormbot>

We bind the value/onChange data handlers once and they get automatically set for all the gadgets you render. You are only required to set a configKey on each data gadget so that we know where to store that data. Many of our gadget configs, validation UIs, and progressive disclosure UIs use static formbot to simplify their code. When you see Gadgets of type SFBGadgets being passed in to the various components up above, this is what those are referring to.

A special note on context

You can pass context into Formbot.Edit/View and it'll make its way into each individual Gadget's Edit/View component. This was added before React's Context was officially supported and recommended. We should prefer using React.createContext + React.useContext over adding more keys to our context object. That way each gadget can opt in to the context they want. The way it's set up right now can lead to some pretty serious performance problems and should be deprecated.

A special note on conditional visibility / progressive disclosure

Conditional Visibility is the ability for the form to show or hide gadgets based on the values entered into other gadgets. For example, you might use a Radios gadget to ask the user if their favorite food is Cordon Bleu, ___, or other. And if they pick other, then and only then you want to show them a Text gadget to find out what other food is their favorite.

In the inital versions of formbot we called this feature Progressive Disclosure. We have since updated that name to Conditional Visibility as it user-tested better. You'll probably still see remnants in the code referring to pd or ProgDisc or Progressive Disclosure. That's what it's talking about.

Glossary

As you look through Formbot code, you'll probably run across these 3 terms quite often. Here's what they mean:

structure

{ template: {}, metaFields: [], integrationFields: [], trashed: [] }

template is a recursive object structure. Note: its formkeys are not prefixed
metaFields is a list of gadgets with meta.-prefixed formkeys
integrationFields is a list of gadgets with integration.-prefixed formkeys
trashed is deprecated. Move along.

document

{ data, meta, integration }

data is generated as you fill out the form. meta is generated by the server and contains things like submittedBy and workflowSimulation. integration will contain data generated by integrations in workflow.

schema

[gadget]

Each of these gadgets' formkeys are prefixed with one of data., meta., or integration..