CHT Sync provides near real-time data synchronization for dashboards. Discover how this is driving decision-making for healthcare workers and policymakers in Kenya, and how to seamlessly migrate from couch2pg for better data accuracy.
The role of data dashboards in community health
A member of the Community Health Toolkit (CHT) community, Philip Oyier, recently shared a quote with me on the power of data dashboards.
Dante Vitagliano, who founded Trailmapper, once called data visualization the “language of decision-making.” By his logic, if “good charts effectively convey information”, then “great charts enable, inform, and improve decision-making.”
Now consider what this means for community health workers (CHWs), who gather data as they administer care in hard-to-reach areas where there may otherwise be no formal healthcare system.
Then imagine the policy-makers who rely on the data CHWs generate to decide where resources are needed most, track patient progress, and predict and mitigate trends or anomalies such as potential disease outbreaks.
In healthcare, where timing and targeted action can mean the difference between life and death, these data-driven decisions require reliable and accurate dashboards.
This is where CHT Sync comes in.
Introducing CHT Sync: A game-changer for near real-time data
In countries with digital health programs like Kenya, CHWs use the CHT to collect data while administering care at the community level. As the CHT is offline-first, the CHWs periodically sync to the database for storage. This data is then processed and modeled before being sent to a data visualization tool such as Superset or Metabase.
The data collected on the CHT is stored on the CouchDB database. However, the stored data format on CouchDB cannot be easily manipulated for data management and visualization. Instead, the data has to be forwarded to PostgreSQL, another database where it can be manipulated for visualization.
In recent months, a team at Medic has been working on a solution to make this data visualization near real-time. The new feature, CHT Sync, leverages multiple technologies to integrate the two databases, achieving data synchronization in near real-time as shown below:
“CHT Sync provides real-time data feed to dashboards, facilitating real-time decision making,” Philip Oyier concluded.
From couch2pg to CHT Sync: what’s new?
CHT Sync will directly replace the standalone couch2pg, which users reported had delayed syncing times. CHT Sync addresses this problem by separating the syncing process from the data transformation, which is handled instead by a data building tool called DBT.
“CHT Sync needs to be consistent, fast, and built using industry-standard solutions with long-term support,” explains Diana Bârsan, Principal Software Developer.
“After some exploration, we found that there are no third-party tools that fit our criteria for data synchronization with CouchDB as a source. After almost a decade of using CouchDB and PostgreSQL, we were confident that our expertise allowed us to write such a tool – so we decided to rebuild couch2pg and integrate it into CHT Sync.
“DBT is the industry standard for data transformation. It’s highly customizable, very well documented, and reliable. We found that we can easily migrate existing workflows, as well as build a library of base models to serve as starting points for subsequent transformation that are tailored to specific CHT configurations.”
Software Developer Tom Wier shared how CHT Sync also addresses the issue of outdated data in PostgreSQL.
“It’s important to know if the PostgreSQL data is out of date, by how much, and why. To make this easier to observe, CHT Sync includes key metrics within PostgreSQL, and a service to send these to a Prometheus server. These metrics track liveness of the syncing service, and the delays at each stage.
Together, they give a full picture of PostgreSQL’s data status, reassurance that it is up to date, and troubleshooting clues if it’s not.”
CHT Sync: What impact is it having in Kenya?
CHT Sync is production ready! Earlier this year we worked on migrating dashboards in Kenya’s electronic Community Health Information System (eCHIS) over to it.
This has had a tremendous impact. Users now have the latest streamed data from CHWs, facilitating data-driven discussions that can create more effective resource allocation, streamlined processes, and better overall care coordination.
Ultimately, by transforming complex data into actionable insights, reliable dashboards empower healthcare organizations to optimize their operations and enhance the quality of care delivered to patients.
Lorena Rodriguez, our Quality Assurance Engineer, found: “Through quality assurance and end-to-end testing, we have significantly improved the reliability and timeliness of data with CHT Sync.
“We have made the synchronization process more robust to enhance data accuracy and minimize latency, meaning users can be confident that the data they are seeing is both current and comprehensive. Rigorous end-to-end testing validates its consistency across the system. Moving forward, a benchmarking analysis will compare CHT CouchDB and CHT Sync, providing critical insights to further optimize performance.”
Streamlining migration to CHT Sync: Lessons learned
The migration of these dashboards was a learning experience for the team. Software Developer Njuguna Ndung’u explained:
“This migration was a good opportunity to rethink the database framework. It might seem easier to translate the existing tables and views into DBT models, but that can easily lead to a lot more work as you try and optimize the performance of the queries.
“Instead, starting from the dashboard and what needs to be shown, then working backwards to what data is required and from where, is a much better approach. Medic has provided base tables that users can build on top of to query the required data.”
Supporting developers: Simplified setup and documentation
For community-based health projects that still need to migrate their dashboards, we have made the process as straightforward as possible by creating a set of instructions and learnings for users on the CHT doc site.
Key considerations include setting up CHT Sync with either Docker Compose or Kubernetes as the containerization technologies, accounting for environmental variables, building the DBT models for data transformation, and creating replica dashboards.
“Feedback so far has highlighted that most users familiar with the CHT have found the setup process easy and straightforward when following the documentation, with a few encountering minor issues that were quickly resolved with the support of the team,” our UX Researcher, Ziithe Hiwa, shared.
“This has allowed us to refine our documentation to ensure developers are able to set up CHT Sync more independently. Preliminary findings have shown that it improves dashboard responsiveness, allowing users to access timely, high-quality data.”
If you’re ready to set up CHT Sync, reach out to @Ziithe on the CHT forum or open an issue on the CHT Sync repository.
