How Data Mapping Works

DHIS2-, OpenMRS- and FHIR Integration all use the ValueSource class to map CommCare data to API resources.

A ValueSource is given in JSON format. e.g.

{
  "case_property": "active",
  "jsonpath": "$.active"
}

This ValueSource maps the value from the case property named “active” to the “active” property of an API resource.

Different Sources of Values

The ValueSource class supports several different sources of values:

  • case_property: As seen above, a ValueSource can be used for fetching a value from a case property, or setting a value on a case property.

  • form_question: Fetches a value from a form question. e.g. “/data/foo/bar” will get the value of a form question named “bar” in the group “foo”. Form metadata is also available, e.g. “/metadata/received_on” is the server time when the form submission was received. You can find more details in the source code at corehq.motech.value_source:FormQuestion

  • case_owner_ancestor_location_field: Specifies a location metadata field name. The ValueSource will start at the location of the case owner, traverse up their location hierarchy, and return the first value it finds for a location with that field. This can be used for mapping CommCare locations to locations or organization units in a remote system.

  • form_user_ancestor_location_field: Specifies a location metadata field name. Similar to case_owner_ancestor_location_field but for forms instead of cases. The ValueSource will start at the location of the user who submitted the form, traverse up their location hierarchy, and return the first value it finds for a location with that field. This can be used for mapping CommCare locations to locations or organization units in a remote system.

  • subcase_value_source: Defines a ValueSource to be evaluated on the subcases of a case. e.g.

    {
      "subcase_value_source": {"case_property": "name"}
      "case_type": "child",
      "is_closed": false,
      "jsonpath": "$.childrensNames"
    }
    
  • supercase_value_source: Defines a ValueSource to be evaluated on the parent/host case of a case. e.g.

    {
      "supercase_value_source": {"case_property": "name"}
      "referenced_type": "mother",
      "jsonpath": "$.mothersName"
    }
    
  • value: A constant value. This can be used for exporting a constant, or it can be combined with case_property for importing a constant value to a case property. See corehq.motech.value_source:ConstantValue for more details.

Data Types

Integrating structured data with remote systems can involve converting data from one format or data type to another. Use data type declarations to cast the data type of a value.

For standard OpenMRS properties (person properties, name properties and address properties) MOTECH will set data types correctly, and integrators do not need to worry about them.

But administrators may want a value that is a date in CommCare to a datetime in a remote system, or vice-versa. To convert from one to the other, set data types for value sources.

The default is for both the CommCare data type and the external data type not to be set. e.g.

{
  "expectedDeliveryDate": {
    "case_property": "edd",
    "commcare_data_type": null,
    "external_data_type": null
  }
}

To set the CommCare data type to a date and the OpenMRS data type to a datetime for example, use the following:

{
  "expectedDeliveryDate": {
    "case_property": "edd",
    "commcare_data_type": "cc_date",
    "external_data_type": "omrs_datetime"
  }
}

For the complete list of CommCare data types, see MOTECH constants. For the complete list of DHIS2 data types, see DHIS2 constants. For the complete list of OpenMRS data types, see OpenMRS constants.

Import-Only and Export-Only Values

In configurations like OpenMRS Atom feed integration that involve both sending data to OpenMRS and importing data from OpenMRS, sometimes some values should only be imported, or only exported.

Use the direction property to determine whether a value should only be exported, only imported, or (the default behaviour) both.

For example, to import a patient value named “hivStatus” as a case property named “hiv_status” but not export it, use "direction": "in":

{
  "hivStatus": {
    "case_property": "hiv_status",
    "direction": "in"
  }
}

To export a form question, for example, but not import it, use "direction": "out":

{
  "hivStatus": {
    "case_property": "hiv_status",
    "direction": "out"
  }
}

Omit direction, or set it to null, for values that should be both imported and exported.

Getting Values From JSON Documents

JSONPath has emerged as a standard for navigating JSON documents. It is supported by PostgreSQL, SQL Server, and others. ValueSource uses it to read values from JSON API resources.

And, in the case of FHIR Integration, it also uses it to build FHIR resources.

See the article by Stefan Goessner, who created JSONPath, for more details.

OpenMRS observations and Bahmni diagnoses can be imported as extension cases of CommCare case. This is useful for integrating patient referrals, or managing diagnoses.

Values from the observation or diagnosis can be imported to properties of the extension case. MOTECH needs to traverse the JSON response from the remote system in order to get the right value. Value sources can use JSONPath to do this.

Here is a simplified example of a Bahmni diagnosis to get a feel for JSONPath:

{
  "certainty": "CONFIRMED",
  "codedAnswer": {
    "conceptClass": "Diagnosis",
    "mappings": [
      {
        "code": "T68",
        "name": "Hypothermia",
        "source": "ICD 10 - WHO"
      }
    ],
    "shortName": "Hypothermia",
    "uuid": "f7e8da66-f9a7-4463-a8ca-99d8aeec17a0"
  },
  "creatorName": "Eric Idle",
  "diagnosisDateTime": "2019-10-18T16:04:04.000+0530",
  "order": "PRIMARY"
}

The JSONPath for “certainty” is simply “certainty”.

The JSONPath for “shortName” is “codedAnswer.shortName”.

The JSONPath for “code” is “codedAnswer.mappings[0].code”.

For more details, see How to Inspect an Observation or a Diagnosis in the documentation for the MOTECH OpenMRS & Bahmni Module.

The value_source Module

class corehq.motech.value_source.CaseOwnerAncestorLocationField(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, case_owner_ancestor_location_field: str)[source]

A reference to a location metadata value. The location may be the case owner, the case owner’s location, or the first ancestor location of the case owner where the metadata value is set.

e.g.

{
  "doc_type": "CaseOwnerAncestorLocationField",
  "location_field": "openmrs_uuid"
}
__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, case_owner_ancestor_location_field: str) None

Method generated by attrs for class CaseOwnerAncestorLocationField.

classmethod wrap(data)[source]

Allows us to duck-type JsonObject, and useful for doing pre-instantiation transforms / dropping unwanted attributes.

class corehq.motech.value_source.CaseProperty(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, case_property: str)[source]

A reference to a case property value.

e.g. Get the value of a case property named “dob”:

{
  "case_property": "dob"
}
__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, case_property: str) None

Method generated by attrs for class CaseProperty.

class corehq.motech.value_source.CasePropertyConstantValue(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, value: str, value_data_type: str = 'cc_text', case_property: str)[source]
__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, value: str, value_data_type: str = 'cc_text', case_property: str) None

Method generated by attrs for class CasePropertyConstantValue.

class corehq.motech.value_source.ConstantValue(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, value: str, value_data_type: str = 'cc_text')[source]

ConstantValue provides a ValueSource for constant values.

value must be cast as value_data_type.

get_value() returns the value for export. Use external_data_type to cast the export value.

get_import_value() and deserialize() return the value for import. Use commcare_data_type to cast the import value.

>>> one = ConstantValue.wrap({
...     "value": 1,
...     "value_data_type": COMMCARE_DATA_TYPE_INTEGER,
...     "commcare_data_type": COMMCARE_DATA_TYPE_DECIMAL,
...     "external_data_type": COMMCARE_DATA_TYPE_TEXT,
... })
>>> info = CaseTriggerInfo("test-domain", None)
>>> one.deserialize("foo")
1.0
>>> one.get_value(info)  # Returns '1.0', not '1'. See note below.
'1.0'

Note

one.get_value(info) returns '1.0', not '1', because get_commcare_value() casts value as commcare_data_type first. serialize() casts it from commcare_data_type to external_data_type.

This may seem counter-intuitive, but we do it to preserve the behaviour of ValueSource.serialize().

__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, value: str, value_data_type: str = 'cc_text') None

Method generated by attrs for class ConstantValue.

deserialize(external_value: Any) Any[source]

Converts the value’s external data type or format to its data type or format for CommCare, if necessary, otherwise returns the value unchanged.

class corehq.motech.value_source.FormQuestion(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, form_question: str)[source]

A reference to a form question value.

e.g. Get the value of a form question named “bar” in the group “foo”:

{
  "form_question": "/data/foo/bar"
}

Note

Normal form questions are prefixed with “/data”. Form metadata, like “received_on” and “userID”, are prefixed with “/metadata”.

The following metadata is available:

Name

Description

deviceID

An integer that identifies the user’s device

timeStart

The device time when the user opened the form

timeEnd

The device time when the user completed the form

received_on

The server time when the submission was received

username

The user’s username without domain suffix

userID

A large unique number expressed in hexadecimal

instanceID

A UUID identifying this form submission

__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, form_question: str) None

Method generated by attrs for class FormQuestion.

class corehq.motech.value_source.FormUserAncestorLocationField(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, form_user_ancestor_location_field: str)[source]

A reference to a location metadata value. The location is the form user’s location, or the first ancestor location of the form user where the metadata value is set.

e.g.

{
  "doc_type": "FormUserAncestorLocationField",
  "location_field": "dhis_id"
}
__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, form_user_ancestor_location_field: str) None

Method generated by attrs for class FormUserAncestorLocationField.

classmethod wrap(data)[source]

Allows us to duck-type JsonObject, and useful for doing pre-instantiation transforms / dropping unwanted attributes.

class corehq.motech.value_source.SubcaseValueSource(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, subcase_value_source: dict, case_types: Optional[List[str]] = None, is_closed: Optional[bool] = None)[source]

A reference to a list of child/extension cases.

Evaluates nested ValueSource config, allowing for recursion.

__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, subcase_value_source: dict, case_types: Optional[List[str]] = None, is_closed: Optional[bool] = None) None

Method generated by attrs for class SubcaseValueSource.

set_external_value(external_data, info)[source]

Builds external_data by reference.

Currently implemented for dicts using JSONPath but could be implemented for other types as long as they are mutable.

class corehq.motech.value_source.SupercaseValueSource(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, supercase_value_source: dict, identifier: Optional[str] = None, referenced_type: Optional[str] = None, relationship: Optional[str] = None)[source]

A reference to a list of parent/host cases.

Evaluates nested ValueSource config, allowing for recursion.

__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None, supercase_value_source: dict, identifier: Optional[str] = None, referenced_type: Optional[str] = None, relationship: Optional[str] = None) None

Method generated by attrs for class SupercaseValueSource.

set_external_value(external_data, info)[source]

Builds external_data by reference.

Currently implemented for dicts using JSONPath but could be implemented for other types as long as they are mutable.

class corehq.motech.value_source.ValueSource(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None)[source]

Subclasses model a reference to a value, like a case property or a form question.

Use the get_value() method to fetch the value using the reference, and serialize it, if necessary, for the external system that it is being sent to.

__init__(*, external_data_type: Optional[str] = None, commcare_data_type: Optional[str] = None, direction: Optional[str] = None, value_map: Optional[dict] = None, jsonpath: Optional[str] = None) None

Method generated by attrs for class ValueSource.

deserialize(external_value: Any) Any[source]

Converts the value’s external data type or format to its data type or format for CommCare, if necessary, otherwise returns the value unchanged.

get_value(case_trigger_info: corehq.motech.value_source.CaseTriggerInfo) Any[source]

Returns the value referred to by the ValueSource, serialized for the external system.

serialize(value: Any) Any[source]

Converts the value’s CommCare data type or format to its data type or format for the external system, if necessary, otherwise returns the value unchanged.

set_external_value(external_data: dict, info: corehq.motech.value_source.CaseTriggerInfo)[source]

Builds external_data by reference.

Currently implemented for dicts using JSONPath but could be implemented for other types as long as they are mutable.

classmethod wrap(data: dict)[source]

Allows us to duck-type JsonObject, and useful for doing pre-instantiation transforms / dropping unwanted attributes.

corehq.motech.value_source.deserialize(value_source_config: jsonobject.containers.JsonDict, external_value: Any) Any[source]

Converts the value’s external data type or format to its data type or format for CommCare, if necessary, otherwise returns the value unchanged.

corehq.motech.value_source.get_case_location(case)[source]

If the owner of the case is a location, return it. Otherwise return the owner’s primary location. If the case owner does not have a primary location, return None.

corehq.motech.value_source.get_form_question_values(form_json)[source]

Given form JSON, returns question-value pairs, where questions are formatted “/data/foo/bar”.

e.g. Question “bar” in group “foo” has value “baz”:

>>> get_form_question_values({'form': {'foo': {'bar': 'baz'}}})
{'/data/foo/bar': 'baz'}
corehq.motech.value_source.get_import_value(value_source_config: jsonobject.containers.JsonDict, external_data: dict) Any[source]

Returns the external value referred to by the value source definition, deserialized for CommCare.

corehq.motech.value_source.get_value(value_source_config: jsonobject.containers.JsonDict, case_trigger_info: corehq.motech.value_source.CaseTriggerInfo) Any[source]

Returns the value referred to by the value source definition, serialized for the external system.