You spend hours crafting feedback after reviewing an article. You want the author to understand and avoid repeating the mistakes. Then you see the same issues in their next submission.
That’s precisely what happened to us while working on a client’s community writing program. We would spend hours reviewing content, crafting clear feedback, and ensuring our tone remained constructive. However, authors continued to make similar mistakes despite receiving detailed explanations.
This led us to build an AI feedback assistant. The goal was to help us craft clear and effective feedback while maintaining relationships with authors and saving time.
The results were immediate. Review sessions that previously took over two hours now take just thirty minutes.
Here’s how we did it.
The Problem
After timing over twenty content reviews for a client’s community writing program, we discovered something surprising. Creating professional feedback takes three times as much time as identifying technical issues.
Reading through and identifying issues took just ten to twenty minutes. But crafting the feedback? That took one to two hours.
Professional writers typically wouldn’t need extensive corrections. However, in community writing programs, most writers are technical professionals first and writers second. They’re prone to making recurring mistakes.
Additionally, external writers lack the same context as internal team members. Without the right tone, feedback can sound harsh or impolite. This could discourage future contributions.
We needed to make the feedback process more efficient. We also had to ensure that feedback remained clear, professional, and effective.
Why Asking ChatGPT Won’t Work
The obvious approach seems straightforward: “Just ask ChatGPT to improve your feedback.”
We tried this. It didn’t work.
Basic improvement prompts gave us several problems:
- Generic feedback that sounded robotic and missed nuanced context
- Inconsistent tone, varying wildly in professionalism and directness
- Inconsistent length, either too verbose or too concise, never hitting the right balance
The output still needed extensive editing.
We wanted something different. We needed a tool that consistently generated feedback requiring minimal editing. Something we could feed a quick comment like “this part isn’t clear” and receive complete, professional feedback in return. We also wanted to dictate long, rambling thoughts and get back something concise and sharp.
We needed an intentional approach.
Our Approach: Solving the Problem in 5 Steps
To solve this problem systematically, we broke it down into five steps:
- Requirements specification (defining the output)
- AI interaction design (defining the input)
- AI model testing and selection
- Prompt engineering
- Workflow integration
Requirements Specification: Defining the Output
The first step involved defining our requirements. We needed to know what effective feedback should look like.
We identified five criteria that feedback needed to meet:
- Clear problem identification: Authors must understand what the problem is. This way, they can not only fix the issue but also prevent it from happening again. Effective feedback must clearly state what specific issue needs to be addressed.
- Actionable solutions: Writers need to know how to fix an issue. For specific problems, such as grammar or word choice, the feedback assistant provides direct corrections. For broader issues, it offers suggestions without being overly specific. This gives authors autonomy over their work so they still feel in control of their piece.
- Appropriate length: Too short, and the feedback lacks clarity. Too long, and the feedback becomes overwhelming. The feedback assistant needs to strike the right balance.
- Professional tone: We wanted to encourage authors to keep contributing to our client’s community writing program. Feedback needed to offer constructive criticism using a professional and collaborative tone.
- Human-like quality: Feedback that sounds artificial could cause authors to feel like they’re receiving generic responses. This could discourage future contributions. The feedback needed to sound natural and conversational.
These five criteria provided a clear framework for effective feedback.
With a clear picture of our desired output, the next step was defining how to interact with the AI.
Input: How We Interact with the AI
We needed a system that could capture raw thoughts and produce clear feedback.
Sometimes we might jot down something as brief as “this part isn’t clear.” We expected the AI to generate complete, professional feedback that meets all our requirements. Other times, we might dictate long, rambling thoughts about multiple issues. We needed the AI to organize and condense these into concise, effective communication.
This meant the AI needed to understand our specific context and standards. It couldn’t just apply generic “good feedback” principles. It had to know our style guide, understand the technical domain we work in, and grasp the relationship dynamics of community writing programs.
With our input requirements clear, we needed to choose the right AI model for the job.
Choosing the AI Model
We chose our AI model based on human-like quality.
Since natural-sounding feedback was a major requirement, we needed a model that could produce conversational feedback. For that, we chose Claude Sonnet 4.
We tested several options, including GPT-4, which would do an equally good job. However, we went with Claude since we use it for most of our writing tasks. It produces responses that sound human more consistently.
After choosing our model, the next step was engineering the system prompt.
Prompt Engineering
Specificity and context are everything when writing effective system prompts.
You can’t just tell the model, “make the feedback concise.” How concise are we talking about? Two sentences? Three? Four?
The more specific you are in your instructions and context, the more likely you are to get what you want.
To give the AI model specific context and instructions, we gathered data from previous review sessions. We collected examples of good and bad feedback, analyzing them to identify their characteristics. This analysis became detailed instructions and context for the model.
To ensure we covered edge cases we might have missed in our instructions, we used few-shot prompting. This technique involves providing the AI with selected examples of both good and bad feedback from our data. We used the rest of our examples for evaluation.
With our prompt ready, we were ready to integrate it into our workflow.
Creating a Claude Project
We created the feedback assistant as a Claude project.
The workflow is straightforward. We paste the article and our raw comments into the Claude interface. It returns polished feedback that meets all our requirements.
The interface looks clean and intuitive. [Here we would show the actual interface rather than a placeholder.]
Simple, but we’ve seen immediate results.
Review sessions that used to take over two hours now take thirty minutes at most. Now we can review more content and work with more writers.
Our next step is to make it work anywhere. Whether we’re on GitHub or Google Docs, the assistant will be able to capture comments and return context-aware feedback.
Should You Build Your Own Feedback Assistant?
Every content team needs an AI feedback assistant.
You can build this yourself. However, this could mean weeks of prompt engineering and testing iterations to get consistent results.
You could invest that time and effort. Or you can get a working solution in a week.
TinyRocket specializes in building AI automation systems for content teams. We implement content automation workflows that speed up your content review process. This helps you create quality content more quickly and consistently.
Ready to remove your content bottlenecks? Book a call. Let’s have a chat.