SQL on FHIR: analytics on form data
Description
45 cells · updated Feb 18, 2025
Load sample data
We generated 100 000 sample responses for the Risk Assessment Questionnaire provided in FHIR Analytics with Aidbox and Metabase sample project.
The following cell loads responses (QuestionnaireResponse resources) and extracted resources (Observation and Patient resources).
POST /v2/fhir/$import
Accept: text/yaml
Content-Type: text/yaml
id: myimp
contentEncoding: gzip
inputs:
- resourceType: Patient
url: https://storage.googleapis.com/aidbox-public/events/2025-02-philippines-bootcamp/patient.ndjson.gz
- resourceType: QuestionnaireResponse
url: https://storage.googleapis.com/aidbox-public/events/2025-02-philippines-bootcamp/questionnaireresponse.ndjson.gz
- resourceType: Observation
url: https://storage.googleapis.com/aidbox-public/events/2025-02-philippines-bootcamp/observation.ndjson.gz{}
Data
The extracted observation are of three kinds:
- Body mass index (BMI);
- Smoking status (smoker, passive smoker, stopped smoking, never smoked);
- Excessive alcohol consumption (yes/no)
Let's look at each observation more closely
BMI
BMI observation contain BMI value and SNOMED code defining observation type.
GET /fhir/Observation/gen-obs-bmi-1
accept: text/yamlmeta:
lastUpdated: '2025-01-28T14:56:39.967857Z'
versionId: '81'
extension:
- {url: 'ex:createdAt', valueInstant: '2025-01-28T14:56:39.967857Z'}
derivedFrom:
- {reference: QuestionnaireResponse/gen-qr-1}
valueQuantity: {value: 24.064332204316088}
resourceType: Observation
effectiveDateTime: '2025-01-28T13:44:37.022833976Z'
status: final
id: gen-obs-bmi-1
code:
coding:
- {code: '60621009', system: 'http://snomed.info/sct', display: 'Body Mass Index '}
Smoking status
Smoking status observation contain SNOMED code 365981007 (Tobacco smoking behavior — finding) defining the observation kind and one of the following SNOMED codes:
- 77176002 (smoker)
- 266919005 (never smoked)
- 160617001 (stopped smoking)
- 43381005 (passive smoker)
GET /fhir/Observation/gen-obs-smoking-1
Accept: text/yamlmeta:
lastUpdated: '2025-01-28T14:56:39.967857Z'
versionId: '81'
extension:
- {url: 'ex:createdAt', valueInstant: '2025-01-28T14:56:39.967857Z'}
derivedFrom:
- {reference: QuestionnaireResponse/gen-qr-1}
resourceType: Observation
effectiveDateTime: '2025-01-28T13:44:37.022833976Z'
status: final
id: gen-obs-smoking-1
code:
coding:
- {code: '365981007', system: 'http://snomed.info/sct', display: Tobacco smoking behavior - finding}
valueCodeableConcept:
coding:
- {code: '77176002', system: 'http://snomed.info/sct', display: Smoker}
Excessive alcohol consumption
Excessive alcohol consumption observation contains SNOMED code 719848005 (Disorder caused by alcohol) defining observation kind and one of the following SNOMED codes as value:
- 373067005 (no)
- 373066001 (yes)
Note that the questionnaire contains question: "Have you ever consumed alcohol?". And the observation is created only if alcohol has ever been consumed.
GET /fhir/Observation/gen-obs-exc-alc-2{
"meta": {
"lastUpdated": "2025-01-28T14:56:39.967857Z",
"versionId": "81",
"extension": [
{
"url": "ex:createdAt",
"valueInstant": "2025-01-28T14:56:39.967857Z"
}
]
},
"derivedFrom": [
{
"reference": "QuestionnaireResponse/gen-qr-2"
}
],
"resourceType": "Observation",
"effectiveDateTime": "2025-01-28T13:44:37.022833976Z",
"status": "final",
"id": "gen-obs-exc-alc-2",
"code": {
"coding": [
{
"code": "719848005",
"system": "http://snomed.info/sct",
"display": "Disorder caused by alcohol"
}
]
},
"valueCodeableConcept": {
"coding": [
{
"code": "373067005",
"system": "http://snomed.info/sct",
"display": "No"
}
]
}
}Create ViewDefinition resources
As we have seen above, Observation resources are quite nested and inconvenient to use in analytical SQL queries.
SQL on FHIR provides a way to extract relevant data in nice tabular format: ViewDefinition resources.
In our case all Observation resources have similar structure: each of them contains observed value and a code defining observation kind.
BMI
We will create a table with the following structure
| Observation id | BMI value | QuestionnaireResponse id |
|---|
Note that Aidbox uses string type by default, therefore we need to specify datatype for the BMI value.
PUT /fhir/ViewDefinition/observation_bmi_view
Content-Type: text/yaml
Accept: text/yaml
name: observation_bmi_view
resource: observation
select:
- column:
- name: observation_id
path: getResourceKey()
- name: bmi
path: value.ofType(Quantity).value
type: decimal
- name: qr_id
path: derivedFrom.first().getReferenceKey()
where:
- path: code.coding.where(code = '60621009').exists()
status: active
id: observation_bmi_view
resourceType: ViewDefinitionwhere:
- {path: code.coding.where(code = '60621009').exists()}
meta:
lastUpdated: '2025-02-17T13:20:38.778225Z'
versionId: '107'
extension:
- {url: 'https://fhir.aidbox.app/fhir/StructureDefinition/created-at', valueInstant: '2025-02-17T13:20:38.778225Z'}
name: observation_bmi_view
resourceType: ViewDefinition
status: active
id: observation_bmi_view
resource: observation
select:
- column:
- {name: observation_id, path: getResourceKey()}
- {name: bmi, path: value.ofType(Quantity).value, type: decimal}
- {name: qr_id, path: derivedFrom.first().getReferenceKey()}
Aidbox automatically create a view in the sof schema. Let's look at data.
select * from sof.observation_bmi_view
limit 10| bmi | qr_id | observation_id |
|---|---|---|
| 24.064332204316088 | gen-qr-1 | gen-obs-bmi-1 |
| 23.320073211330087 | gen-qr-10 | gen-obs-bmi-10 |
| 30.760773314666764 | gen-qr-100 | gen-obs-bmi-100 |
| 24.825445454831456 | gen-qr-1000 | gen-obs-bmi-1000 |
| 27.455811393648887 | gen-qr-10000 | gen-obs-bmi-10000 |
| 21.7619619118027 | gen-qr-100000 | gen-obs-bmi-100000 |
| 23.233307017944394 | gen-qr-10001 | gen-obs-bmi-10001 |
| 28.6701388177837 | gen-qr-10002 | gen-obs-bmi-10002 |
| 28.247785615346345 | gen-qr-10003 | gen-obs-bmi-10003 |
| 32.117852778104975 | gen-qr-10004 | gen-obs-bmi-10004 |
Smoking status
This is similar to the previous case.
We will create a table with the following structure
| Observation id | SNOMED code | Human-readable value | QuestionnaireResponse id |
|---|
PUT /fhir/ViewDefinition/observation_smoking_view
Content-Type: text/yaml
Accept: text/yaml
name: observation_smoking_view
resource: observation
select:
- column:
- name: observation_id
path: getResourceKey()
- name: code
path: value.ofType(CodeableConcept).coding.first().code
- name: display
path: value.ofType(CodeableConcept).coding.first().display
- name: qr_id
path: derivedFrom.first().getReferenceKey()
where:
- path: code.coding.where(code = '365981007').exists()
status: active
id: observation_smoking_view
resourceType: ViewDefinitionwhere:
- {path: code.coding.where(code = '365981007').exists()}
meta:
lastUpdated: '2025-02-17T13:20:47.141847Z'
versionId: '110'
extension:
- {url: 'https://fhir.aidbox.app/fhir/StructureDefinition/created-at', valueInstant: '2025-02-17T13:20:47.141847Z'}
name: observation_smoking_view
resourceType: ViewDefinition
status: active
id: observation_smoking_view
resource: observation
select:
- column:
- {name: observation_id, path: getResourceKey()}
- {name: code, path: value.ofType(CodeableConcept).coding.first().code}
- {name: display, path: value.ofType(CodeableConcept).coding.first().display}
- {name: qr_id, path: derivedFrom.first().getReferenceKey()}
select * from sof.observation_smoking_view
limit 10;| code | qr_id | display | observation_id |
|---|---|---|---|
| 77176002 | gen-qr-1 | Smoker | gen-obs-smoking-1 |
| 266919005 | gen-qr-10 | Never smoked tobacco | gen-obs-smoking-10 |
| 77176002 | gen-qr-100 | Smoker | gen-obs-smoking-100 |
| 266919005 | gen-qr-1000 | Never smoked tobacco | gen-obs-smoking-1000 |
| 160617001 | gen-qr-10000 | Stopped smoking | gen-obs-smoking-10000 |
| 266919005 | gen-qr-100000 | Never smoked tobacco | gen-obs-smoking-100000 |
| 160617001 | gen-qr-10001 | Stopped smoking | gen-obs-smoking-10001 |
| 77176002 | gen-qr-10002 | Smoker | gen-obs-smoking-10002 |
| 43381005 | gen-qr-10003 | Passive smoker | gen-obs-smoking-10003 |
| 43381005 | gen-qr-10004 | Passive smoker | gen-obs-smoking-10004 |
Excessive alcohol consumption observation
This is the same as smoking status, only observation kind code differs.
| Observation id | SNOMED code | Human-readable value | QuestionnaireResponse id |
|---|
PUT /fhir/ViewDefinition/observation_alcohol_view
Content-Type: text/yaml
Accept: text/yaml
name: observation_alcohol_view
resource: observation
select:
- column:
- name: observation_id
path: getResourceKey()
- name: code
path: value.ofType(CodeableConcept).coding.first().code
- name: display
path: value.ofType(CodeableConcept).coding.first().display
- name: qr_id
path: derivedFrom.first().getReferenceKey()
where:
- path: code.coding.where(code = '719848005').exists()
status: active
id: observation_alcohol_view
resourceType: ViewDefinitionwhere:
- {path: code.coding.where(code = '719848005').exists()}
meta:
lastUpdated: '2025-02-17T13:21:02.093064Z'
versionId: '113'
extension:
- {url: 'https://fhir.aidbox.app/fhir/StructureDefinition/created-at', valueInstant: '2025-02-17T13:21:02.093064Z'}
name: observation_alcohol_view
resourceType: ViewDefinition
status: active
id: observation_alcohol_view
resource: observation
select:
- column:
- {name: observation_id, path: getResourceKey()}
- {name: code, path: value.ofType(CodeableConcept).coding.first().code}
- {name: display, path: value.ofType(CodeableConcept).coding.first().display}
- {name: qr_id, path: derivedFrom.first().getReferenceKey()}
SELECT * FROM sof.observation_alcohol_view
LIMIT 10| code | qr_id | display | observation_id |
|---|---|---|---|
| 373066001 | gen-qr-100 | Yes | gen-obs-exc-alc-100 |
| 373067005 | gen-qr-1000 | No | gen-obs-exc-alc-1000 |
| 373067005 | gen-qr-10000 | No | gen-obs-exc-alc-10000 |
| 373066001 | gen-qr-10001 | Yes | gen-obs-exc-alc-10001 |
| 373067005 | gen-qr-10002 | No | gen-obs-exc-alc-10002 |
| 373066001 | gen-qr-10008 | Yes | gen-obs-exc-alc-10008 |
| 373067005 | gen-qr-10009 | No | gen-obs-exc-alc-10009 |
| 373066001 | gen-qr-10011 | Yes | gen-obs-exc-alc-10011 |
| 373067005 | gen-qr-10012 | No | gen-obs-exc-alc-10012 |
| 373067005 | gen-qr-10013 | No | gen-obs-exc-alc-10013 |
QuestionnaireResponse
We will take gender into account in our analytical queries. It is not extracted from the sample QuestionnaireResponse resource, so we will create ViewDefinition to extract it.
| QuestionnaireResponse id | gender |
|---|
PUT /fhir/ViewDefinition/qr_view
Content-Type: text/yaml
Accept: text/yaml
name: qr_view
resource: questionnaireresponse
select:
- column:
- name: qr_id
path: getResourceKey()
- name: gender
path: >-
item.where(linkId='Zicv0gIs').item.where(linkId='Jw_kMVir').answer.first().value.ofType(Coding).code
where:
- path: questionnaire = 'http://forms.aidbox.io/questionnaire/cvd-form'
status: active
id: qr_view
resourceType: ViewDefinitionwhere:
- {path: 'questionnaire = ''http://forms.aidbox.io/questionnaire/cvd-form'''}
meta:
lastUpdated: '2025-02-17T13:21:12.436424Z'
versionId: '116'
extension:
- {url: 'https://fhir.aidbox.app/fhir/StructureDefinition/created-at', valueInstant: '2025-02-17T13:21:12.436424Z'}
name: qr_view
resourceType: ViewDefinition
status: active
id: qr_view
resource: questionnaireresponse
select:
- column:
- {name: qr_id, path: getResourceKey()}
- {name: gender, path: item.where(linkId='Zicv0gIs').item.where(linkId='Jw_kMVir').answer.first().value.ofType(Coding).code}
select * from sof.qr_view limit 10;| qr_id | gender |
|---|---|
| gen-qr-1 | male |
| gen-qr-10 | female |
| gen-qr-100 | male |
| gen-qr-1000 | male |
| gen-qr-10000 | male |
| gen-qr-100000 | female |
| gen-qr-10001 | male |
| gen-qr-10002 | female |
| gen-qr-10003 | male |
| gen-qr-10004 | male |
Creating custom views
SQL on FHIR v2 describes how to flatten a single resource. It is useful to create custom views for our cases.
First, let's create a schema for our views
CREATE SCHEMA IF NOT EXISTS u_analytics;Next views will prepare data for visualization
Combined smoking data
Let's join smoking observations view and questionnaire response view to get combined data, then transform machine-readable codes to human-readable text.
CREATE OR REPLACE VIEW u_analytics.smoking AS
SELECT
qr.gender AS gender,
CASE
WHEN obs.code = '77176002' THEN 'Smoker'
WHEN obs.code = '266919005' THEN 'Never smoked'
WHEN obs.code = '160617001' THEN 'Stopped smoking'
WHEN obs.code = '43381005' THEN 'Passive smoker'
END AS status
FROM sof.observation_smoking_view obs
JOIN sof.qr_view qr USING (qr_id);Result:
SELECT * FROM u_analytics.smoking LIMIT 10;| gender | status |
|---|---|
| male | Smoker |
| female | Never smoked |
| male | Smoker |
| male | Never smoked |
| male | Stopped smoking |
| female | Never smoked |
| male | Stopped smoking |
| female | Smoker |
| male | Passive smoker |
| male | Passive smoker |
Combined alcohol view
Alcohol observations are absent if a person never consumed alcohol. For our analytics it is useful to have this information present.
CREATE OR REPLACE VIEW u_analytics.alcohol AS
SELECT
qr.gender AS gender,
CASE
WHEN code = '373067005' THEN 'Not binge drinker'
WHEN code = '373066001' THEN 'Binge drinker'
ELSE 'Not drinker'
END AS status
FROM sof.observation_alcohol_view obs
RIGHT JOIN sof.qr_view qr USING (qr_id);Result:
SELECT * FROM u_analytics.alcohol LIMIT 10;| gender | status |
|---|---|
| male | Not drinker |
| female | Not drinker |
| male | Binge drinker |
| male | Not binge drinker |
| male | Not binge drinker |
| female | Not drinker |
| male | Binge drinker |
| female | Not binge drinker |
| male | Not drinker |
| male | Not drinker |
Combined BMI view
And here it is useful to have both BMI and assessment (overweight or not)
CREATE OR REPLACE VIEW u_analytics.bmi AS
SELECT
qr.gender AS gender,
CASE
WHEN obs.bmi >= 25 THEN 'Overweight or obese'
ELSE 'Normal weight'
END status,
obs.bmi as bmi
FROM sof.observation_bmi_view obs
JOIN sof.qr_view qr USING (qr_id);Result:
SELECT * FROM u_analytics.bmi LIMIT 10;| bmi | gender | status |
|---|---|---|
| 24.064332204316088 | male | Normal weight |
| 23.320073211330087 | female | Normal weight |
| 30.760773314666764 | male | Overweight or obese |
| 24.825445454831456 | male | Normal weight |
| 27.455811393648887 | male | Overweight or obese |
| 21.7619619118027 | female | Normal weight |
| 23.233307017944394 | male | Normal weight |
| 28.6701388177837 | female | Overweight or obese |
| 28.247785615346345 | male | Overweight or obese |
| 32.117852778104975 | male | Overweight or obese |
Analytics
Smoking statistics
What proportion of adults smoke (by sex and overall)?
select
gender,
status,
count(*) * 100 / sum(count(*)) over () AS "percentage (total)",
count(*) * 100 / sum(count(*)) over (PARTITION BY gender) as "percentage (group)"
FROM u_analytics.smoking
GROUP BY (status, gender)
ORDER BY gender asc, "percentage (group)" desc
limit 10;| gender | status | percentage (group) | percentage (total) |
|---|---|---|---|
| female | Passive smoker | 38.153463758819754 | 19.034 |
| female | Never smoked | 29.14728992944195 | 14.541 |
| female | Smoker | 22.799069916613213 | 11.374 |
| female | Stopped smoking | 9.90017639512508 | 4.939 |
| male | Passive smoker | 38.022828863346106 | 19.054 |
| male | Never smoked | 28.997046615581098 | 14.531 |
| male | Smoker | 23.060344827586206 | 11.556 |
| male | Stopped smoking | 9.91977969348659 | 4.971 |
Excessive alcohol consumption
What proportion of adult drank excessively in the past 6 months (by sex and overall)?
Note that we use RIGHT JOIN here, since not all responses generate observation, they don't if a person never drank.
select
gender,
status,
count(*) * 100 / sum(count(*)) over () AS "percentage (total)",
count(*) * 100 / sum(count(*)) over (PARTITION BY gender) as "percentage (group)"
FROM u_analytics.alcohol
GROUP BY (status, gender)
ORDER BY gender asc, "percentage (group)" desc
limit 10;| gender | status | percentage (group) | percentage (total) |
|---|---|---|---|
| female | Not binge drinker | 41.89183771648493 | 20.899 |
| female | Not drinker | 32.831542655548425 | 16.379 |
| female | Binge drinker | 25.276619627966646 | 12.61 |
| male | Not binge drinker | 41.54693486590038 | 20.82 |
| male | Not drinker | 33.13777139208174 | 16.606 |
| male | Binge drinker | 25.31529374201788 | 12.686 |
Unhealthy BMI
What proportion of adults are overweight or obese (BMI > 25), by gender and overall?
select
gender,
status,
count(*) * 100 / sum(count(*)) over () AS "percentage (total)",
count(*) * 100 / sum(count(*)) over (PARTITION BY gender) as "percentage (group)"
FROM u_analytics.bmi
GROUP BY (status, gender)
ORDER BY gender asc, "percentage (group)" desc
limit 10;| gender | status | percentage (group) | percentage (total) |
|---|---|---|---|
| female | Overweight or obese | 53.39761064785119 | 26.639 |
| female | Normal weight | 46.60238935214881 | 23.249 |
| male | Overweight or obese | 73.65501277139208 | 36.91 |
| male | Normal weight | 26.34498722860792 | 13.202 |