I gave a talk at the Edmonton Data Society recently about how modern data platforms support real public sector workflows. I opened it by saying something that probably sounded like false modesty: I'm not a senior architect. I'm someone who builds pipelines and reports most days, and this is what I've learned from getting my hands dirty.
That framing turned out to matter, because the most useful thing I've learned has very little to do with technology. Below is the argument, minus the live demo.
Policy runs on data, and data lives in many places
Start with how a decision actually gets made in government. A policy maker needs a timely, reliable answer — how much funding went where, and did it reach the people it was meant for. That answer has to be assembled from systems that were each built for a different purpose: one for applications, one for enrollments, one for approvals, one for payments.
Here's the part that gets misdiagnosed. Those systems aren't broken. Each one serves its own team well — that's why it exists and why it's still running. The data already exists, and it's mostly fine where it sits. The opportunity isn't replacing any of it. It's bringing it together so the combined picture is consistent, trusted, and ready to support a decision.
This distinction sounds academic until you watch a modernization project frame itself as a rip-out and replace. Those projects consume years and goodwill. The ones that work build a connective layer on top of systems that keep doing their jobs.
The problem is rarely the technology
When reporting is slow or contested, the instinct is to look for a technical culprit. In my experience the friction distributes across three areas, and only one of them is technical.
People: one metric, several readings
Ask three teams for enrollment and you may get three numbers, all defensible, all computed slightly differently. Nobody is wrong. There just isn't a shared definition. So meetings that were supposed to be about a decision become meetings about whose figure is right, and the real work happens after everyone leaves.
Process: the same preparation, every cycle
Reporting typically spans several steps and several teams. Each cycle repeats work that was done last cycle — the same exports, the same reconciliation, the same cleanup. It's reliable in the sense that it eventually produces a number, but a new question can take weeks to answer because the path to the answer has to be rebuilt by hand.
Technology: a distributed landscape
The data sits across systems and formats. Connecting them takes deliberate effort. This is the piece everyone focuses on, and it's real — but it's also the most tractable of the three. Buying a platform solves some of this. It solves none of the first two on its own.
What "good" looks like
Before evaluating any platform, it's worth writing down what you actually need it to do. My list has five items, and I'd encourage testing every vendor claim against them rather than the other way around:
- Centralized access. One consistent layer for data from every system — no duplicates, no version drift.
- Scalable processing. Growing volumes and multiple teams without resource contention or queuing behind each other.
- Standardized transformation. Cleaning logic defined once — repeatable, reviewable, and inspectable by someone who didn't write it.
- Governed and secure access. Roles, privacy-aligned controls, and full auditability, with every query traceable.
- Ready for consumption. Curated datasets that BI tools connect to directly, producing consistent metrics by construction.
Notice that four of the five are about discipline rather than horsepower. The platform matters, but mostly as the place where that discipline gets enforced automatically instead of depending on everyone remembering.
Governance is the part that makes data defensible
This is the argument I care most about, and the one I think gets undersold in vendor material because it isn't exciting.
Governance usually gets framed as a constraint — the compliance tax you pay for having data. I'd argue the opposite. Governance is what converts a dashboard into something you can put in front of a minister, an auditor, or a journalist and defend line by line. It comes down to three questions a platform should answer structurally:
- Who sees what? Role-based and row-level access so a ministry analyst sees everything while an institution sees only its own rows — enforced by the platform, not by policy documents and good intentions.
- What stays protected? Masking so sensitive values are hidden from roles without clearance. Privacy alignment becomes configuration rather than a manual review step someone can forget.
- Can you defend it? Access history and the ability to reproduce a report exactly as it ran on the day it was published. When someone asks "how was this number produced?", the answer is a query, not an archaeology project.
That asymmetry is why I think public sector data work is genuinely harder than the private sector equivalent, and why "move fast" translates badly here. The bar isn't whether the number is right. It's whether you can demonstrate it was right, months later, to someone who is not inclined to take your word for it.
The honest part about cost
Consumption-based platforms are usually sold on the upside: you pay for what you use, storage is cheap, no capital purchase, compute scales on demand. All true, and it genuinely fits public procurement better than buying hardware on a five-year cycle.
The part that gets less airtime is that costs run away when compute is left running or oversized. The same elasticity that lets you scale up for a heavy job will happily bill you for an idle one. Auto-suspend, right-sizing, and hard credit ceilings all exist — but they are settings, not defaults you can assume. Someone has to own them, and that someone should be named before the first invoice, not after.
Where AI fits
I get asked about this in every session, so let me be direct about my position.
Clean data, a governed model, and traceable lineage are the prerequisite for AI, not an alternative to it. You cannot bolt intelligence onto a mess and expect the output to be trustworthy — you'll have automated the production of confident wrong answers. Every organization I've seen get value from AI on operational data had done the unglamorous foundation work first.
When the foundation is there, the credible use cases tend to be narrow: forecasting demand, flagging anomalies in payments, letting a non-technical user query a well-modelled dataset in plain language. Narrow beats grand in the public sector, because narrow is something you can evaluate, explain, and roll back.
And governance doesn't retire when AI arrives. AI outputs still need to be traceable, auditable, and policy-aligned — arguably more so, since the reasoning is harder to inspect. The access controls you build for reporting should apply to AI workloads running against the same data, inside the same perimeter.
What adoption actually takes
The sequence that works is unglamorous: pilot one dataset and one report to prove value in weeks rather than quarters. Then move a single real reporting process end to end, so the platform carries something that matters. Then expand onto a governed foundation that already exists.
And the friction to plan for is mostly not technical. Data residency and privacy review are a path, not a blocker — but only if security is in the room early rather than presented with a finished design. Integration beats replacement, since the new platform has to sit alongside what teams already use. Skills are rarely the constraint, because teams that already write SQL are most of the way there.
The hardest part is trust and adoption. A platform nobody uses is a line item, not a capability. Win one team properly and let the result do the arguing.
What I'd want you to take away
- Start with the problem, not the tool. Credible solutions come from understanding how data supports a decision inside real constraints — not from a capability list.
- One platform should carry the whole path. Source to raw to cleaned to modelled to dashboard. Every handoff between disconnected tools is a place where trust leaks out.
- Governance is not optional. Access control, masking, lineage, and audit are the difference between a dashboard and a defensible, policy-grade data product.
- Value lives in decisions, not dashboards. The end state isn't a report anyone admires. It's a better funding call, made because the data was ready, governed, and trusted.
That last one is the whole thing, really. It's easy to measure output — how many dashboards shipped, how many sources onboarded. It's harder, and more honest, to ask whether any decision changed because of them.
Adapted from a session delivered to the Edmonton Data Society. The views here are my own and don't represent my employer. If you're working through similar questions in your own organization, I'm always glad to compare notes — get in touch.
← Back to all articles