How Data Mapping Works

DHIS2-, OpenMRS- and FHIR Integration all use the ValueSource class to map CommCare data to API resources.

A ValueSource is given in JSON format. e.g.

{
  "case_property": "active",
  "jsonpath": "$.active"
}

This ValueSource maps the value from the case property named “active” to the “active” property of an API resource.

Different Sources of Values

The ValueSource class supports several different sources of values:

case_property: As seen above, a ValueSource can be used for fetching a value from a case property, or setting a value on a case property.
form_question: Fetches a value from a form question. e.g. “/data/foo/bar” will get the value of a form question named “bar” in the group “foo”. Form metadata is also available, e.g. “/metadata/received_on” is the server time when the form submission was received. You can find more details in the source code at corehq.motech.value_source:FormQuestion
case_owner_ancestor_location_field: Specifies a location metadata field name. The ValueSource will start at the location of the case owner, traverse up their location hierarchy, and return the first value it finds for a location with that field. This can be used for mapping CommCare locations to locations or organization units in a remote system.
form_user_ancestor_location_field: Specifies a location metadata field name. Similar to case_owner_ancestor_location_field but for forms instead of cases. The ValueSource will start at the location of the user who submitted the form, traverse up their location hierarchy, and return the first value it finds for a location with that field. This can be used for mapping CommCare locations to locations or organization units in a remote system.

subcase_value_source: Defines a ValueSource to be evaluated on the subcases of a case. e.g.

{
  "subcase_value_source": {"case_property": "name"}
  "case_type": "child",
  "is_closed": false,
  "jsonpath": "$.childrensNames"
}

supercase_value_source: Defines a ValueSource to be evaluated on the parent/host case of a case. e.g.

{
  "supercase_value_source": {"case_property": "name"}
  "referenced_type": "mother",
  "jsonpath": "$.mothersName"
}

value: A constant value. This can be used for exporting a constant, or it can be combined with case_property for importing a constant value to a case property. See corehq.motech.value_source:ConstantValue for more details.

Data Types

Integrating structured data with remote systems can involve converting data from one format or data type to another. Use data type declarations to cast the data type of a value.

For standard OpenMRS properties (person properties, name properties and address properties) MOTECH will set data types correctly, and integrators do not need to worry about them.

But administrators may want a value that is a date in CommCare to a datetime in a remote system, or vice-versa. To convert from one to the other, set data types for value sources.

The default is for both the CommCare data type and the external data type not to be set. e.g.

{
  "expectedDeliveryDate": {
    "case_property": "edd",
    "commcare_data_type": null,
    "external_data_type": null
  }
}

To set the CommCare data type to a date and the OpenMRS data type to a datetime for example, use the following:

{
  "expectedDeliveryDate": {
    "case_property": "edd",
    "commcare_data_type": "cc_date",
    "external_data_type": "omrs_datetime"
  }
}

For the complete list of CommCare data types, see MOTECH constants. For the complete list of DHIS2 data types, see DHIS2 constants. For the complete list of OpenMRS data types, see OpenMRS constants.

Import-Only and Export-Only Values

In configurations like OpenMRS Atom feed integration that involve both sending data to OpenMRS and importing data from OpenMRS, sometimes some values should only be imported, or only exported.

Use the direction property to determine whether a value should only be exported, only imported, or (the default behaviour) both.

For example, to import a patient value named “hivStatus” as a case property named “hiv_status” but not export it, use "direction": "in":

{
  "hivStatus": {
    "case_property": "hiv_status",
    "direction": "in"
  }
}

To export a form question, for example, but not import it, use "direction": "out":

{
  "hivStatus": {
    "case_property": "hiv_status",
    "direction": "out"
  }
}

Omit direction, or set it to null, for values that should be both imported and exported.

Getting Values From JSON Documents

JSONPath has emerged as a standard for navigating JSON documents. It is supported by PostgreSQL, SQL Server, and others. ValueSource uses it to read values from JSON API resources.

And, in the case of FHIR Integration, it also uses it to build FHIR resources.

See the article by Stefan Goessner, who created JSONPath, for more details.

OpenMRS observations and Bahmni diagnoses can be imported as extension cases of CommCare case. This is useful for integrating patient referrals, or managing diagnoses.

Values from the observation or diagnosis can be imported to properties of the extension case. MOTECH needs to traverse the JSON response from the remote system in order to get the right value. Value sources can use JSONPath to do this.

Here is a simplified example of a Bahmni diagnosis to get a feel for JSONPath:

{
  "certainty": "CONFIRMED",
  "codedAnswer": {
    "conceptClass": "Diagnosis",
    "mappings": [
      {
        "code": "T68",
        "name": "Hypothermia",
        "source": "ICD 10 - WHO"
      }
    ],
    "shortName": "Hypothermia",
    "uuid": "f7e8da66-f9a7-4463-a8ca-99d8aeec17a0"
  },
  "creatorName": "Eric Idle",
  "diagnosisDateTime": "2019-10-18T16:04:04.000+0530",
  "order": "PRIMARY"
}

The JSONPath for “certainty” is simply “certainty”.

The JSONPath for “shortName” is “codedAnswer.shortName”.

The JSONPath for “code” is “codedAnswer.mappings[0].code”.

For more details, see How to Inspect an Observation or a Diagnosis in the documentation for the MOTECH OpenMRS & Bahmni Module.

The value_source Module

class corehq.motech.value_source.CaseOwnerAncestorLocationField(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, case_owner_ancestor_location_field: str)[source]

A reference to a location metadata value. The location may be the case owner, the case owner’s location, or the first ancestor location of the case owner where the metadata value is set.

e.g.

{
  "doc_type": "CaseOwnerAncestorLocationField",
  "location_field": "openmrs_uuid"
}

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, case_owner_ancestor_location_field: str) → None: Method generated by attrs for class CaseOwnerAncestorLocationField.

classmethod wrap(data)[source]: Allows us to duck-type JsonObject, and useful for doing pre-instantiation transforms / dropping unwanted attributes.

class corehq.motech.value_source.CaseProperty(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, case_property: str)[source]

A reference to a case property value.

e.g. Get the value of a case property named “dob”:

{
  "case_property": "dob"
}

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, case_property: str) → None: Method generated by attrs for class CaseProperty.

class corehq.motech.value_source.CasePropertyConstantValue(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, value: str, value_data_type: str = 'cc_text', case_property: str)[source]

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, value: str, value_data_type: str = 'cc_text', case_property: str) → None: Method generated by attrs for class CasePropertyConstantValue.

class corehq.motech.value_source.ConstantValue(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, value: str, value_data_type: str = 'cc_text')[source]

ConstantValue provides a ValueSource for constant values.

value must be cast as value_data_type.

get_value() returns the value for export. Use external_data_type to cast the export value.

get_import_value() and deserialize() return the value for import. Use commcare_data_type to cast the import value.

>>> one = ConstantValue.wrap({
...     "value": 1,
...     "value_data_type": COMMCARE_DATA_TYPE_INTEGER,
...     "commcare_data_type": COMMCARE_DATA_TYPE_DECIMAL,
...     "external_data_type": COMMCARE_DATA_TYPE_TEXT,
... })
>>> info = CaseTriggerInfo("test-domain", None)
>>> one.deserialize("foo")
1.0
>>> one.get_value(info)  # Returns '1.0', not '1'. See note below.
'1.0'

Note

one.get_value(info) returns '1.0', not '1', because get_commcare_value() casts value as commcare_data_type first. serialize() casts it from commcare_data_type to external_data_type.

This may seem counter-intuitive, but we do it to preserve the behaviour of ValueSource.serialize().

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, value: str, value_data_type: str = 'cc_text') → None: Method generated by attrs for class ConstantValue.

deserialize(external_value: Any) → Any[source]: Converts the value’s external data type or format to its data type or format for CommCare, if necessary, otherwise returns the value unchanged.

class corehq.motech.value_source.FormQuestion(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, form_question: str)[source]

A reference to a form question value.

e.g. Get the value of a form question named “bar” in the group “foo”:

{
  "form_question": "/data/foo/bar"
}

Note

Normal form questions are prefixed with “/data”. Form metadata, like “received_on” and “userID”, are prefixed with “/metadata”.

The following metadata is available:

Name	Description
deviceID	An integer that identifies the user’s device
timeStart	The device time when the user opened the form
timeEnd	The device time when the user completed the form
received_on	The server time when the submission was received
username	The user’s username without domain suffix
userID	A large unique number expressed in hexadecimal
instanceID	A UUID identifying this form submission

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, form_question: str) → None: Method generated by attrs for class FormQuestion.

class corehq.motech.value_source.FormUserAncestorLocationField(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, form_user_ancestor_location_field: str)[source]

A reference to a location metadata value. The location is the form user’s location, or the first ancestor location of the form user where the metadata value is set.

e.g.

{
  "doc_type": "FormUserAncestorLocationField",
  "location_field": "dhis_id"
}

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, form_user_ancestor_location_field: str) → None: Method generated by attrs for class FormUserAncestorLocationField.

classmethod wrap(data)[source]: Allows us to duck-type JsonObject, and useful for doing pre-instantiation transforms / dropping unwanted attributes.

class corehq.motech.value_source.SubcaseValueSource(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, subcase_value_source: dict, case_types: List[str] | None = None, is_closed: bool | None = None)[source]

A reference to a list of child/extension cases.

Evaluates nested ValueSource config, allowing for recursion.

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, subcase_value_source: dict, case_types: List[str] | None = None, is_closed: bool | None = None) → None: Method generated by attrs for class SubcaseValueSource.

set_external_value(external_data, info)[source]

Builds external_data by reference.

Currently implemented for dicts using JSONPath but could be implemented for other types as long as they are mutable.

A reference to a list of parent/host cases.

Evaluates nested ValueSource config, allowing for recursion.

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None, supercase_value_source: dict, identifier: str | None = None, referenced_type: str | None = None, relationship: str | None = None) → None: Method generated by attrs for class SupercaseValueSource.

set_external_value(external_data, info)[source]

Builds external_data by reference.

Currently implemented for dicts using JSONPath but could be implemented for other types as long as they are mutable.

Subclasses model a reference to a value, like a case property or a form question.

Use the get_value() method to fetch the value using the reference, and serialize it, if necessary, for the external system that it is being sent to.

__init__(*, external_data_type: str | None = None, commcare_data_type: str | None = None, direction: str | None = None, value_map: dict | None = None, jsonpath: str | None = None) → None: Method generated by attrs for class ValueSource.

deserialize(external_value: Any) → Any[source]: Converts the value’s external data type or format to its data type or format for CommCare, if necessary, otherwise returns the value unchanged.

get_value(case_trigger_info: CaseTriggerInfo) → Any[source]: Returns the value referred to by the ValueSource, serialized for the external system.

serialize(value: Any) → Any[source]: Converts the value’s CommCare data type or format to its data type or format for the external system, if necessary, otherwise returns the value unchanged.

set_external_value(external_data: dict, info: CaseTriggerInfo)[source]

Builds external_data by reference.

Currently implemented for dicts using JSONPath but could be implemented for other types as long as they are mutable.

classmethod wrap(data: dict)[source]: Allows us to duck-type JsonObject, and useful for doing pre-instantiation transforms / dropping unwanted attributes.

corehq.motech.value_source.deserialize(value_source_config: JsonDict, external_value: Any) → Any[source]: Converts the value’s external data type or format to its data type or format for CommCare, if necessary, otherwise returns the value unchanged.

corehq.motech.value_source.get_case_location(case)[source]: If the owner of the case is a location, return it. Otherwise return the owner’s primary location. If the case owner does not have a primary location, return None.

corehq.motech.value_source.get_form_question_values(form_json)[source]

Given form JSON, returns question-value pairs, where questions are formatted “/data/foo/bar”.

e.g. Question “bar” in group “foo” has value “baz”:

>>> get_form_question_values({'form': {'foo': {'bar': 'baz'}}})
{'/data/foo/bar': 'baz'}

corehq.motech.value_source.get_import_value(value_source_config: JsonDict, external_data: dict) → Any[source]: Returns the external value referred to by the value source definition, deserialized for CommCare.

corehq.motech.value_source.get_value(value_source_config: JsonDict, case_trigger_info: CaseTriggerInfo) → Any[source]: Returns the value referred to by the value source definition, serialized for the external system.