The Suite

An application’s suite.xml file controls its structure.

The full XML spec for the suite is availabe on the commcare-core wiki.

Overview

Suite generation starts with Application.create_suite, which delegates to SuiteGenerator.

Suite generation is organized based on its major XML elements: resources, entries, details, etc. The suite is generated in two passes:

corehq.apps.app_manager.suite_xml.sections generates independendent parts. A “section” is one of the major elements that goes into the suite: resources, entries, details, etc. This logic relies on the app document itself.
corehq.apps.app_manager.suite_xml.post_process handles logic that depends on the first pass being complete. Some of this logic adds new elements, some manipulates existing elements. This logic relies on the app document and also on the XML models generated by the first pass. Anything that deals with stacks must be a post processor, to guarantee that all menus have already been generated.

Challenges for developers in suite generation code:

Language mixes CommCare concepts, such as “datum” and “menu”, with HQ concepts, such as “modules”
Lots of branching
Has evolved one feature at a time, sometimes without attention to how different features interact
CommCare’s suite spec supports plenty of behavior that HQ doesn’t allow the app builder to configure. In some areas, 20% of the HQ logic handles 80% of what’s actually supported, so the code is more complex than the developer might expect. As an example of this, the suite code generally supports an arbitrary number of datums per form, even though the vast majority of forms only require one or two cases.

A bright spot: test coverage for suite generation is good, and adding new tests is typically straightforward.

Sections

DetailContributor

Details represent the configuration for case lists and case details. The reuse of the word “Detail” here is unfortunate. Details can be used for other purposes, such as the referral_detail, but 99% of the time they’re used for case list/detail.

The case list is the “short” detail and the case detail is the “long” detail. A handful of configurations are only supported for one of these, e.g., actions only get added to the short detail.

The detail element can be nested. HQ never nests short details, but it nests long details to produce tabbed case details. Each tab has its own <detail> element.

The bulk of detail configuration is in the display properties, called “fields” and sometimes “columns” in the code. Each field has a good deal of configuration, and the code transforms them into named tuples while processing them. Each field has a format, one of about a dozen options. Formats are typically either UI-based, such as formatting a phone number to display as a link, or calculation-based, such as configuring a property to display differently when it’s “late”, i.e., is too far past some reference date.

Most fields map to a particular case property, with the exception of calculated properties. These calculated properties are identified only by number. A typical field might be called case_dob_1 in the suite, indicating both its position and its case property, but a calculation would be called case_calculated_property_1.

EntriesContributor

This is the largest and most complex of the suite sections, responsible for generating an <entry> element for each form, including the datums required for form entry. The EntriesHelper, which does all of the heavy lifting here, is imported into other places in HQ that need to know what datums a form requires, such as the session schema generator for form builder and the UI for form linking.

When forms work with multiple datums, they need to be named in a way that is predictable for app builders, who reference them inside forms. This is most relevant to the “select parent first” feature and to parent/child modules. See update_refs and rename_other_id, both inner functions in add_parent_datums, plus this comment on matching parent and child datums.

FixtureContributor

This contributor adds a tiny fixture with a demo user group.

It’s also the parent class for SchedulerFixtureContributor, a flagged feature.

MenuContributor

Menus approximately correspond to HQ modules.

Menus almost correspond to command lists, the screens in CommCare that ask the user to select a form or sub-menu. However, if the suite contains multiple <menu> elements with the same id, they will be concatenated and displayed as a single screen.

Menu ids will typically map to the module’s position in the application: the first menu is m0, second is m1, etc.

Highlights of menu configuration:

Display conditions, which become relevant attributes
Display-only forms, which becomes the put_in_root attribute
Grid style, to determine whether the command list should be displayed as a flat list or as a grid that emphasizes the menu icons

Resource Contributors

These contributors let the suite know where to find external resources,. These external resources are text files that are also part of the application’s CCZ.

FormResourceContributor handles XForms
LocaleResourceContributor handles the text files containing translations
PracticeUserRestoreContributor handles a dummy restore used for Practice Mode

Post Processors

EndpointsHelper

This is support for session endpoints, which are a flagged feature for mobile that also form the basis of smart links in web apps.

Endpoints define specific locations in the application using a stack, so they rely on similar logic to end of form navigation. The complexity of generating endpoints is all delegated to WorkflowHelper.

InstancesHelper

Every instance referenced in an xpath expression needs to be added to the relevant entry or menu node, so that CommCare knows what data to load when. This includes case list calculations, form/menu display conditions, assertions, etc.

HQ knows about a particular set of instances (locations, reports, etc.). There’s factory-based code dealing with these “known” instances. When a new feature involves any kind of XPath calculation, it needs to be scanned for instances.

Instances are used to reference data beyond the scope of the current XML document. Examples are the commcare session, casedb, lookup tables, mobile reports, case search data etc.

Instances are added into the suite file in <entry> or <menu> elements and directly in the form XML. This is done in post processing of the suite file in corehq.apps.app_manager.suite_xml.post_process.instances.

How instances work

When running applications instances are initialized for the current context using an instance declaration which ties the instance ID to the actual instance model:

<instance id=”my-instance” ref=”jr://fixture/my-fixture” />

This allows using the fixture with the specified ID:

instance(‘my-instance’)path/to/node

From the mobile code point of view the ID is completely user defined and only used to ‘register’ the instance in current context. The index ‘ref’ is used to determine which instance is attached to the given ID.

Instances in CommCare HQ

In CommCare HQ we allow app builders to reference instance in many places in the application but don’t require that the app builder define the full instance declaration.

When ‘building’ the app we rely on instance ID conventions to enable the build process to determine what ‘ref’ to use for the instances used in the app.

For static instances like ‘casedb’ the instance ID must match a pre-defined name. For example

casedb
commcaresession
groups

Other instances use a namespaced convention: “type:sub-type”. For example:

commcare-reports:<uuid>
item-list:<fixture name>

Custom instances

App builders can define custom instances in a form using the ‘CUSTOM_INSTANCES’ plugin

RemoteRequestsHelper

The <remote-request> descends from the <entry>. Remote requests provide support for CommCare to request data from the server and then allow the user to select an item from that data and use it as a datum for a form. In practice, remote requests are only used for case search and claim workflows.

This case search config UI in app manager is a thin wrapper around the various elements that are part of <remote-request>, which means RemoteRequestsHelper is not especially complicated, although it is rather long.

Case search and claim is typically an optional part of a workflow. In this use case, the remote request is accessed via an action, and the rewind construct is used to go back to the main flow. However, the flag USH_INLINE_SEARCH supports remote requests being made in the main flow of a session. When using this flag, a <post> and query datums are added to a normal form <entry>. This makes search inputs available after the search, rather than having them destroyed by rewinding.

This module includes SessionEndpointRemoteRequestFactory, which generates remote requests for use by session endpoints. This functionality exists for the sake of smart links: whenever a user clicks a smart link, any cases that are part of the smart link need to be claimed so the user can access them.

ResourceOverrideHelper

This is dead code. It supports a legacy feature, multi-master linked applications.

The actual flag has been removed, but a lot of related code still exists.

WorkflowHelper

This is primarily used for end of form navigation and form linking. It contains logic to determine the proper sequence of commands to navigate a particular place in an app, such as a specific case list. It also needs to provide any datums required to reach that place in the app.

Because CommCare’s UI logic is driven by the data currently in the user’s session and the data needed by forms, rather than being directly configured, this means HQ needs to predict how CommCare’s UI logic will behave, which is difficult and results in code that’s easily disturbed by new features that influence navigation.

Understanding stacks in the CommCare Session is useful for working with WorkflowHelper.

Some areas to be aware of:

Datums can either require manual selection (from a case list) or can be automatically selected (such as the usercase id).
HQ names each datum, defaulting to case_id for datums selected from case lists. When HQ determines that a form requires multiple datums, it creates a new id for the new datum, which will often incorporate the case type. It also may need to rename datums that already exist - see _replace_session_references_in_stack.
To determine which datums are distinct and which represent the same piece of information, HQ has matching logic in _find_best_match.
get_frame_children generates the list of frame children that will navigate to a given form or module, mimicking CommCare’s navigation logic
Shadow modules complicate this entire area, because they use their source module’s forms but their own module configuration.
There are a bunch of advanced features with their own logic, such as advanced modules, but even the basic logic is fairly complex.
Within end of form navigation and form linking, the “previous screen” option is the most fragile. Form linking has simpler code, since it pushes the complexity of the feature onto app builders.