If a government agency uses software to decide whether your family needs a closer look, you have a right to ask how that software works. A University of Iowa study, as reported by Phys, argues that social service agencies using AI need firm safeguards. This guide translates the study's recommendations into plain terms and into questions you can ask, whether you are a parent, a caregiver, or someone who works with these systems.
The story: AI is entering public services
Per Phys, the University of Iowa research looks at AI in child welfare, intimate partner violence, and elder abuse cases. Its concern is that these tools can lead to biased or harmful decisions without protections in place. The reporting points to Iowa's new child welfare platform, built in partnership with Google and called VISION, as a real-world example of how quickly AI is arriving in public-service systems.
I can't tell you from the reporting how VISION works internally, so this article does not judge it. The point is that the questions below apply to any system like it.
How "neutral" data becomes biased automation
An AI tool learns from past records. If those records reflect who was already being watched, the tool can repeat that pattern. The study's framing, per Phys, is that data can reproduce existing surveillance of poor families and families of color unless safeguards are built in.
Here is a simple way to picture it. Suppose a family that relies on public benefits appears in more government databases than a wealthier family does. More records can mean more chances to be flagged, even if the two families' circumstances are similar. The software isn't being malicious. It is repeating what the data shows, and the data has gaps and skews.
That is the idea behind "biased automation": an old unfairness, now faster and harder to see.
The stakes
Phys notes that algorithmic decisions can affect family separation, policing, benefits, and access to support. These are not minor outcomes, which is why the study argues for guardrails rather than blind trust.
Four safeguards the study recommends
The recommendations reported by Phys include the following. Each one is something you can look for.
1. Input from survivors and affected people
People who have lived through these systems should help shape them. When you hear about a new agency tool, ask whether people with direct experience were consulted before it launched.
2. Bias audits
A bias audit checks whether a tool treats groups differently, for example by income or race. Ask whether audits happen, how often, and whether results are shared publicly.
3. Decisions people can contest
If an algorithm contributes to a decision about you, there should be a way to challenge it. Ask what the appeal process is, who reviews it, and whether a human can override the tool.
4. Resources over surveillance
The study also recommends using AI to connect families with resources rather than to increase surveillance. Same technology, very different effect: one version points a family to help, the other adds scrutiny.
Surveillance tool or resource matcher?
When you evaluate any AI program in social services, this is the cleanest test. Ask what the tool is for.
- A resource-matching tool helps identify which supports a family might qualify for, such as services and assistance.
- A surveillance tool focuses on flagging risk and monitoring.
Many systems may blend the two. What matters is which purpose drives the design and what happens to a family after the software produces a result.
What to do if you think a system flagged you
The study is about agency policy, not individual how-to steps, so treat the following as practical common sense built on its recommendations rather than legal advice.
- Ask directly. Ask the caseworker whether any software or scoring tool played a part in the decision.
- Ask for the reason in plain language. You are entitled to ask what information led to the outcome.
- Ask how to contest it. Request the appeal or review process in writing if you can.
- Check your records. If the agency can share what is on file, look for errors or outdated details.
- Keep your own notes. Write down dates, names, and what you were told.
- Seek help. A local legal aid office or a family advocate can explain your options where you live.
If you work in or near these agencies
Caregivers and professionals can press for the same safeguards from the inside. Ask these before a tool is adopted:
- Who was consulted, including people affected by the system?
- Has it been audited for bias, and who sees the results?
- Can staff and families challenge its output?
- Is it built to route families to support, or mainly to monitor them?
If nobody can answer, that tells you something.
The takeaway
AI in social services is not automatically help or harm. The Iowa study's message, as Phys reports it, is that the safeguards decide which one you get. Next time you hear about a new algorithm in a public agency, ask four things: who shaped it, who audited it, how can it be challenged, and does it connect people to help or just watch them?
