Egoist Learning
Back to Blog

Self Assessment Prior Knowledge Research Notes

Self-Assessment Should Ask Smaller Questions

Beginner, intermediate, and advanced reduce a whole topic to one global self-rating. Across 20 user tests, questions about specific abilities made learners slow down and think before answering.

By Egoist Labs, Team

Let’s say you’re interested in learning Psychology. You have probably seen some version of this while signing up for a course, using a learning app, or joining a mentorship program:

How much do you know about Psychology?

  • Beginner
  • Intermediate
  • Advanced

You choose “Intermediate,” and the form moves on. But what did that answer actually capture? You may know the important terms but struggle to explain them. You may understand a theory but have no idea how to use it. You may know how to read a study while missing the judgment needed to spot a bad conclusion.

You move on, and that one answer may shape the lesson you receive, the advice you hear, or even the course you get placed into. Yet “Intermediate” still leaves everyone guessing about what you can actually do.

If they guess wrong, you repeat something you already know or get pushed into work you cannot handle yet. The form took seconds. The mismatch can waste hours inside the lesson, the advice, or the learning path that follows.

This problem has been bothering us while building Egoist Learning. Before choosing what happens next, we need a useful picture of what the learner already knows. We do not think one overall level gives us enough.

We expected twelve questions to annoy them

We are testing a different way to assess prior knowledge. Egoist Learning turns one topic into 12 concrete abilities. The learner looks at each one and chooses “Confident,” “Familiar,” or “No idea.”

Before our beta launch, we ran 20 user tests on this flow. In one session, the learner chose Intro to Psychology Concepts. We expected the 12 abilities to feel like another form standing between them and the actual learning. We were waiting for the quick clicks, the tired answers, or the look that says, “Can we move on now?”

The pattern kept repeating across the 20 tests. Learners slowed down, talked through what they knew, and carefully considered each ability before choosing an answer. A question like “How much psychology do you know?” demands one rough guess. A concrete ability gave them something they could remember encountering, explaining, or doing.

The question in their head seemed much closer to:

Can I actually do this?

Twenty tests gave us enough evidence to take that behavior seriously. Learners were willing to examine their own knowledge once the topic became small enough to think about. Better input here could mean less guessing when it is time to choose the lesson, give feedback, or decide where someone should begin.

The next proof is accuracy. We still need to compare those answers with what each learner can explain or do. Our conviction is already stronger: self-assessment should ask about one concrete ability at a time.

The same level can hide two different learners

What someone knows about a topic is usually uneven. They may explain one idea clearly, recognize another without understanding it, and be wrong about something else. One level cannot show those differences.

Imagine two people learning psychology. The first can explain several core ideas but struggles to apply them to a case. The second has seen those ideas used at work but cannot explain why they work. Both may call themselves “intermediate,” even though those differences should change what they learn next.

When all of that becomes one level, the useful details disappear. The next lesson, advice, or placement is based on a guess. The learner may repeat what they already know or start work they are not ready for.

A specific ability is easier to judge because it gives the learner something concrete to remember. “How much psychology do you know?” can pull in grades, pride, embarrassment, and every old story the learner carries about being smart or slow. “Can you state one valid takeaway from a short study summary?” points to something they may remember doing.

Put enough of those small judgments together and a pattern begins to appear. The learner may recognize the ideas but struggle to use them. They may perform a familiar task without understanding the reason behind it. That pattern gives us a more useful place to begin. But it still comes from the learner’s own judgment. How much can we trust it?

Confidence can be wrong in both directions

A learner can feel confident and still be wrong. They can also feel unsure and perform well. Their answer may be affected by previous grades, how familiar the words look, or how easily an answer comes to mind.

Research gives us good reason to be careful. A 2023 meta-analysis covering 160 studies found some agreement between student ratings and expert scores, but the match was far from perfect. On average, students rated themselves slightly too highly. A study of chemistry students found that students also considered past experiences, beliefs about their abilities, and test-taking habits when rating their confidence.

Confidence helps us choose what to check first. The learner’s work tells us how much to trust it. A “Confident” response followed by weak work may expose a misconception. Strong work after a “No idea” response may show that the learner knows more than they think. Either result should change what happens next.

How should self-assessment be designed?

It is easy to design self-assessment around the form. Ask one broad question, offer three levels, collect the answer, and move on. The form is quick to finish, but the learner has to squeeze everything they know into one vague label.

The behavior we saw across 20 user tests points to a better design goal. Give learners questions that make them pause, search their memory, and connect the words on the screen with something they have explained, practiced, or done.

Design for the thinking before the click.

For prior knowledge, this means breaking a topic into specific abilities and asking about them one at a time. The same principle can guide other kinds of self-assessment. Ask about a real action, decision, or experience that the learner can compare with their own memory. This gives them something concrete to think about before choosing an answer.

This matters even more with AI tutoring. “Teach me psychology” gives an AI almost nothing about the learner. Answers about specific abilities give it a place to start. The learner’s next explanation, answer, or attempt can then confirm or correct that picture.

Maybe it is time to rethink how we design self-assessment. “How much do you know?” asks the learner to squeeze an entire subject into one label. “Can you do this?” asks them to think about something specific. A useful answer begins with a question the learner can actually think about.

Research notes

These studies shaped the boundaries and questions in this article: