The Statistics Before the Statistics

Realizing It Was Never Just About the Math

Alex Zajichek, Data Scientist

Department of Quantitative Health Sciences
Cleveland Clinic

September 30, 2026

Setting the stage

Hi everyone, my name is Alex Zajichek, I’m currently a data scientist in the Rotroff lab.
I work on all sorts of different projects in different clinical contexts, from predictive modeling, comparative effectiveness, observational statistical analysis, and now a lot more heavily focused on software development, particularly using AI.
If you’ve used/heard of riskcalc.org, which is Mike’s repository of risk calculators, I’m the one currently maintaining that and currently trying to work on ways to improve it.
And this is actually my second stint at CCF. I was hired by Mike back in 2017 and lived here for about 3 years. Ended up moving back to Wisconsin, bounced around a bit, and then Mike and Alex M convinced me to come back on remotely.
So I’ve been back since early 2023, and now I just work alone in my house…in the woods…in Wisconsin. So it’s nice to see people exist in real life instead of just through a camera. This is actually the first time being back in Cleveland in over 5 years.
And so my brief talk today is really just about how my statistical thinking has evolved over the years and how it really started with Mike and how he helped shape that over time.
And there’s an AI picture of me so you can see what LLM’s think I look like.

Data scientist in the Rotroff Lab
Statistical, modeling, and software projects
Second stint in QHS

AI-me

What are these slides about?

An evolution in statistical problem solving and how Dr. Kattan helped shape it.

Starting (not) at the origin

Alright so like I said, Mike hired back in 2017 and I was fresh out of graduate school (also a special shout out to Jim Bena for making that happen).
And I’m using this graph in the upper right to sort of reflect my views and/capabilities over time with respect to balancing statistical theory with real-world problem solving.
When I first started in 2017 I was probably at my peak in terms of pure technical knowledge, statistical theory, etc. Knew all the theorems, could do all the calculus like nothing, but didn’t know that I really didn’t know anything about solving real world problems.
At that point I had the probably common mindset of a freshly trained statistician: “if this situation occurs, run this method; if that situation occurs, run that method”. It was seemingly objective and systematic. You look at the box the problem fits in, run the “correct” procedure, check diagnostics, calculate p-value, and GET conclusion. Statistics was sort of easy with that view and that’s how I started approaching things.
And I remember on roughly my 3rd day Mike sending me an email: “Here’s something to get your feet wet with.” It was some R code to compute a c-index. He wanted me to run it for some new predictions he had and send him the results. “Sure here you go!”. It was something like this: (show some R code/output).
- “That looks backwards”. I was confused. Then he told me the minimum value is .5, you need to flip the labels.
- I didn’t even know what this meant. He clarified it’s like an AUC. “Oh I’ve done that”. But I never had an intuitive understanding of what it was telling us. Just knew higher was “better”.
- So then the next part: is this good? (looks at recommended thresholds). Maybe? But what if it’s better than the current standard of care/decision making process? “Hmm I never thought of it like that” I guess there is not an objectively right answer. It depends on the context.
- This story is the first time I remember really begining to think that statistics is more than objective math.

Peak technical knowledge
“Use this formula, GET answer”
A C-Index of 0.4?

Fresh Graduate → Started at CCF

Gaining perspective

So Mike started teaching me about decision analysis: decision curves, tradeoffs, clinical context, etc.
The point being that it’s not just a matter of if the model is good statistically, but does it actually help make better decisions? Well, that depends on how decisions are currently made. A poor model might be better than the standard if the standard is a coin flip.
We would talk about risk: flipping a coin, is it deterministic? Mike’s view was: “Given we know EVERYTHING about the coin, it’s a 1 or 0.”. We’re limited by our knowledge about the system. That determines probability. (not talking about quantum here).
And then how it applies to patients. Different patients have different risk for different outcomes, AND have different thresholds/tolerances for different decisions.
And so all of these things over time helped be shift my thinking towards the problem first. The implementation, practical, clinical, technical, resource constraints actually come first and can help INFORM the math.

What risk is and how it applies to patients
Decision curves and tradeoffs
Practical contraints inform the math

Fresh Graduate → Started at CCF

Working biostatistician

Seeing the other side

So with a few years of that experience in hand, after moving back to Wisconsin I eventually ended up joining a local health system working more directly in clinical operations in the Quality department, so I got to work with the clinical teams on the day-to-day running of the clinics and hospitals. It was a completely different perspective from the research context.
For example, I had worked on readmissions prediction in the past in the research context, but then being on the frontlines of readmission prevention from an operational perspective was completely different: hospital metrics don’t line up with the HRRP, different metrics use different definitions, etc.
There was not much attention on statistical theory or mathematical rigor or anything like that. Most of the problems and bottlenecks were just around data and systems not talking to one another. You start to see how much value/impact on patient care there is just from basic information in the right place at the right time, not necessarily overly sophisticated stats.
And so this experience really pulled me down that curve to question a lot of the utility of areas of statistics, particularly p-values. I always think back to this example where someone was presenting results for a continuous improvement project and were presenting a p-value and saying it’s “significant”, but you could feel the sort of empty tension in the room because it just wasn’t convincing because you didn’t FEEL it.
And it reminds of this tongue-in-cheek concept of “interoccular trauma” that is described in The Cult of Statistical Significance by Stephen Ziliak and Deidre McCloskey where they basically say that, paraphasing, “a useful result should cause interoccular trauma, or hit you between the eyes”. You just sort of feel it when see it. And I started to realize that p-values aren’t really ever the thing that do that, for me at least.
That led me to a re-evaluation of my statistical philosophy, and started to move towards Bayesian thinking and causal inference as the preferred approach. Actually, I remember listening to the Learning Bayesian Statistics podcast when Frank Harrell was a guest (some of you may have heard of him as the creator of the rms package which has been widely used by our department, and from what I heard him and Mike would have some fun debates about prediction modeling). He actually gave me the epiphany on the “issue” of subjectivity in Bayesian modeling, and convinced me that, actually, traditional methods have tons of assumptions baked in that we implicitly accept. So in the alternative view, directly accounting for what we know about a given process/mechanism (e.g., how data is missing) by directly representing that expert knowledge in the model, in some sense, is actually MORE objective scientifically.
You shift your focus from the DATA to the MODEL being living, breathing thing representing our understanding. You don’t need data to get it off the ground. Your data only comes in to tune things one way or another, but your model is never “done”.

Value in basic reporting + clean data
Insufficiency of p-values (“interocular trauma”)
Changing statistical philosophy

Fresh Graduate → Started at CCF

Working Biostatistician

Clinical Operations

A renewed focus

So when I came back to CCF, I really had a completely different view/approach to solving statistical problems.
Really thinking of the problem and practical end goal first. The decision making processes, mechanism, structure, implementations, constraints, systems, etc. In some ways for “product” focused.
Mike and I started thinking more about software and tools. For example, how can we integrate for more software practices into statistical project execution? (like Azure DevOps and GitHub more formally). We created the QHS GitHub page as a place to promote more centralized collaboration of open source projects released from the department, and open sourced the code from riskcalc.org to provide transparency to the risk calculators on the site.
And one of the last things Mike and I had been delving more into was a bigger picture of this in terms of how we can begin adding more of this technological rigor and sophistication to research processes/workflows themselves. Like using git, DevOps, and things like that as actual parts of the research output or “product”, not just a means to an end. In some ways the resulting publication as only a component or byproduct of that, not necessarily the sole goal.
CODE, TOOLS, ETC. EQUALLY IMPORTANT AS RESEARCH OUTPUT

The math is only a slice
Software, tools, and transparency
Enhancing our research workflow

Fresh Graduate → Started at CCF

Working Biostatistician

Clinical Operations

Rejoined CCF

In the present

So where I’m at now.
I’m spending more time thinking deeper about scientific and statistical projects than I used to. I probably annoy some of my colleagues and collaborators sometimes when I’m always asking things like “why are we doing this even?” or “what is the end goal here?” or the dreaded “if we’re actually going to implement this in a clinical workflow, shouldn’t we get IT involved first?”
I’m also trying to carry on where Mike and I left off. For one, evaluating how we can revamp the intention of what riskcalc.org is for and can do, and enhancing the technical soundness under the hood.
And I’d be remisced if I didn’t mention AI. Starting to slowly integrate it into all the work I do, which fundamentally is changing how we can think about statistical problems.

Spend more time interrogating the why
Revamp of riskcalc.org
Excited about AI

Thanks to Dr. Kattan for setting me up for this journey, always engaging, debating, and helping shape my thinking along the way.

Fresh Graduate → Started at CCF

Working Biostatistician

Clinical Operations

Rejoined CCF

Current State

Thank You!

References

Deirdre McCloskey, Steve Ziliak. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. University of Michigan Press. 2008. https://doi.org/10.3998/mpub.186351 (quote: chapter 5, page 72)
Harrell, F. E., Jr. (2015). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer
Learning Bayesian Statistics episode with Frank Harrell. https://learnbayesstats.com/episode/45-biostats-clinical-trial-design-with-frank-harrell