Shuichiro Ogawa
日本語

Notes · updated 2026-10-11

What Should Creators Be Able to Take Over When Generative AI Does the Making? (Lessons from Aviation Automation Research)

This note rereads, for creative work, a Japanese article that used aviation regulations to ask which skills software engineers should keep when AI agents write their code (Shimitaro 2026).

Contents (8)
  1. The question and the scope of this note
  2. What aviation regulations and research have protected
  3. Whether the same problems arise with generative AI
  4. How AI errors appear in creative work
  5. What taking over means in creative work
  6. Practices available to creators
  7. Where the aviation analogy does not reach
  8. Questions the research has not yet answered

The question and the scope of this note

In an article on Zenn, Shimitaro asked which skills software engineers should retain when they delegate implementation and review to AI coding agents, and examined the question through aviation regulations (Shimitaro 2026). The article’s answer is the capability to understand the situation and take back the work when the automation fails, which the author calls intervention capability, rather than the ability to write everything oneself. Pilots do not usually fly by hand. However, regulators maintain, through training requirements, a state in which pilots can recover the aircraft when the autopilot disengages. The article transferred this idea to software and proposed three practices: creating exercises in which the AI has made a mistake, reading only high-risk diffs by hand, and turning production incidents into training material.

This note applies the same question to creative work, including design, illustration, writing, and video. When generative AI proposes ideas, draws, and writes, what should creators be able to take over? In addition to the FAA documents that the article cited as primary sources, we examined human factors research on automation, research on generative AI and knowledge work, and experiments on generative AI and creativity. The aim was to identify what changes when the idea is moved into creative work. The largest difference between software and creative work lies in how AI errors appear. The practice of juniors when entry-level work disappears is discussed in What Should Juniors Do When Generative AI Takes Away the Entry-Level Work? (Academic Research), and cognitive offloading and learning are discussed in Does Early Use of Generative AI Inhibit the Formation of Thought? A Literature Map of Cognitive Offloading and Learning. This note is not limited to juniors and addresses how experienced creators maintain their judgment while working with AI.

What aviation regulations and research have protected

The starting point is Bainbridge’s “ironies of automation” (Bainbridge 1983). Designers of automated systems leave to the operator the tasks that they cannot think how to automate. As a result, the operator usually monitors the system and takes over only when an abnormal situation exceeds the automation. However, skills that are not used deteriorate (for physical skills, Bainbridge noted that the refinements of gain and timing are lost first), and attention to a source on which very little happens cannot be maintained for more than about half an hour. Bainbridge closed the paper with a final irony: the most successful automated systems, with rare need for manual intervention, may need the greatest investment in operator training.

Later research separated the reasons why taking over is difficult. Endsley and Kiris studied the automation of a navigation task by an expert system and associated slower decisions after the system failed with lower situation awareness (Endsley and Kiris 1995). The shift from active to passive information processing was the most likely cause. Sarter and Woods, drawing on studies of commercial glass cockpits, argued that automation with many modes requires the operator to track which mode the system is in and what it is doing, and that designs that do not support this demand produce mode errors (Sarter and Woods 1995). Parasuraman and Riley associated misuse, or over-reliance on automation, with monitoring failures and decision biases (Parasuraman and Riley 1997). The review by Parasuraman and Manzey concluded that automation bias occurs in both naive and expert participants and cannot be prevented by training or instructions (Parasuraman and Manzey 2010).

Regulators translated these findings into the treatment of manual flight. SAFO 13002 (January 2013) stated that continuous use of autoflight systems does not reinforce a pilot’s knowledge and skills in manual flight and could degrade the ability to recover the aircraft quickly from an undesired state (FAA 2013). It encouraged operators to emphasize manual flight in both line operations and training. SAFO 17007 (2017) described manual flight as the foundation upon which other technical flying skills are built and noted that manual flight involves more than motor skills (FAA 2017). The same document recommended manual flight practice in all approved combinations of automation and stated that the pilot in command should decide, case by case, when to fly manually. As policies to avoid, it listed overly general statements such as “shall never manually fly at night” and quotas for manual flying that are not based on scientific evidence, such as “shall fly manually at least 10 hrs./month.” The report of the PARC and CAST working group raised concerns about over-reliance on automation and the erosion of manual flying skills (PARC/CAST 2013). The report called for an emphasis on monitoring, cross verification, and the allocation of tasks between the pilot flying (PF) and the pilot monitoring (PM).

Empirical research suggests a distinction regarding which skills decline. Casner et al. asked 16 Boeing 747-400 captains and first officers to fly simulator scenarios while the level of automation was varied (Casner et al. 2014). Instrument scanning and aircraft control were reasonably well retained. More frequent and significant problems were observed in cognitive tasks such as tracking position without a map display, deciding which navigational steps came next, and recognizing instrument failures. Performance on these cognitive tasks was associated with how often pilots engaged in task-unrelated thought while the automation was flying. The authors suggested that the retention of the cognitive skills needed for manual flight may depend on how actively pilots remain engaged in supervising the automation. Ebbatson et al. also showed that the very recent flying experience of airline pilots was related to their manual control strategy (Ebbatson et al. 2010).

Whether the same problems arise with generative AI

Researchers of generative AI have begun to frame the problem through this line of work. Sellen and Horvitz took the term “co-pilot” as a starting point and drew four lessons from aviation: diminished vigilance, challenges in transferring control, de-skilling, and misplaced trust (Sellen and Horvitz 2024). One of their recommendations is to consider situations in which it makes more sense for AI systems to monitor humans rather than the other way round, with spelling and grammar checkers given as an example. Simkute et al. grouped the reasons why generative AI can reduce productivity into four categories based on human factors research (Simkute et al. 2025). These are the shift from production to evaluation, unhelpful restructuring of workflows, interruptions by AI suggestions, and task-complexity polarization, in which automation makes easy tasks easier and hard tasks harder. Tankelevitch et al. argued that generative AI imposes high metacognitive demands on users, including self-awareness of task goals, decomposition of tasks, and well-adjusted confidence in one’s evaluation of outputs (Tankelevitch et al. 2024).

The empirical evidence still consists mainly of self-reports and short-term experiments. Lee et al. collected 936 examples of use from 319 knowledge workers and reported that higher confidence that generative AI could perform a task was associated with less critical thinking, whereas higher self-confidence was associated with more critical thinking (Lee et al. 2025). The nature of critical thinking shifted from gathering information to verifying it and from executing tasks to integrating AI responses and stewarding the task. In the experiment by Dell’Acqua et al. with 758 participants, AI users outperformed non-users on tasks within the frontier of AI capabilities, whereas on a task selected to be outside the frontier, AI users were 19% less likely to produce correct solutions (Dell’Acqua et al. 2026). This frontier is uneven and differs between tasks even within the same workflow and at a seemingly similar level of difficulty. A measured decline in skill has been reported in medicine. Budzyń et al. reported that, at four centers in Poland, the adenoma detection rate of colonoscopies performed without AI decreased from 28.4% to 22.4% after AI support was introduced (Budzyń et al. 2025). Because the study was observational, causality was not established, and the authors stated only that continuous exposure to AI might reduce the detection rate.

In design, Shukla et al. analyzed more than 120 blog posts and subreddit discussions by UX practitioners and found concerns about over-reliance, cognitive offloading, and the erosion of design skills alongside optimism that AI reduces repetitive work (Shukla et al. 2025). This study analyzed discourse and did not measure changes in skill. Within the scope of this review, no study was found that measured changes in the judgment of creative practitioners after continued use of AI.

How AI errors appear in creative work

The article noted that when AI makes mistakes in software, almost all of them fall on the silent side, without alarms. Plausible code is merged without errors or warnings. Even so, software errors have a correct state. Whether a CDK diff can delete a production database can be determined by tracing the behavior of CloudFormation.

Errors in creative work are quieter still. Because there is no single correct answer, AI output appears not as an error but as plausible mediocrity. Experiments show that this mediocrity takes two forms.

The first form is fixation in individual ideation. Design fixation is known as a phenomenon in which designers unknowingly repeat features of examples they have seen. Wadinambiarachchi et al. gave 60 participants a visual ideation task and reported that participants who used an AI image generator showed higher fixation on an initial example than participants without inspiration support, and produced fewer ideas with less variety and lower originality (Wadinambiarachchi et al. 2024). Because AI images are highly finished, the starting point of ideation is drawn toward them.

The second form is homogenization across a group. In the experiment by Doshi and Hauser, stories written by writers with access to AI ideas were evaluated as more creative and better written, and the effect was larger among less creative writers (Doshi and Hauser 2024). At the same time, the AI-enabled stories were more similar to each other. In the comparison by Anderson et al. with 36 participants, the diversity of ideas at the individual level did not differ, but different users produced less semantically distinct ideas with ChatGPT, and ChatGPT users felt less responsible for their ideas (Anderson et al. 2024). Because each work improves, creators have no cue that something has gone wrong. Homogenization becomes visible only when the works of many creators are placed side by side.

The effect of AI also differs by level of expertise. In the experiments by Hou et al., AI increased creativity for all users at the ideation stage, but at the execution stage, expert designers spent more time without improving creativity, and only novices continued to benefit (Hou et al. 2025). The authors attributed this to conflicts between the methods of AI and the established routines of experts.

Where, then, does the judgment needed to discern AI options come from? Schön and Wiggins described architectural designing as a cycle of seeing, moving, and seeing, in which the designer draws, looks at the drawing, and decides the next move from what is seen (Schön and Wiggins 1992). In the second seeing, the designer recognizes unintended as well as intended consequences. For the designer, seeing not only registers information but also constructs its meaning. According to this account, the judgment that discerns the quality of options develops through seeing the results of one’s own moves.

What taking over means in creative work

Taken together, these studies suggest that the capacity to take over in creative work differs somewhat in shape from the intervention capability described in the article. The article’s loop consisted of monitoring the automation, recognizing an abnormality, disengaging the automation, taking over by hand, and recovering from the abnormal state. The article argued that recognizing the abnormality is the most difficult step in software. In creative work, the task of defining the abnormality itself remains with the person. Because there is neither an alarm nor a correct answer, it is the creator’s judgment that decides what is mediocre and what has drifted from the intention.

This note defines the capacity to take over in creative work as the following three abilities. Reframing: checking whether the question that AI output answers matches the question of the work, and restating the question when it does not. Discernment: being able to state, by one’s own criteria, the quality of options and where they are mediocre. Redirection: stopping the flow of AI output and producing options in a different direction by one’s own hand. In creative work, these three abilities correspond to the engineer’s recovery from an abnormal state more closely than correcting errors in output does.

The analogy of PM and PF also applies directly to creative work. In aviation, the PM is a person trained to perform as PF, and the article drew on this point to conclude that hands-on practice cannot be abandoned completely. In creative work, this premise matters even more. According to Schön and Wiggins, discernment develops through seeing the results of one’s own moves. Therefore, a monitor who does not move may become unable to update the very criteria used for discernment. The distinction shown by Casner et al., that cognitive skills decline more readily than manual skills, also suggests, when moved into creative work, that creators need to deliberately maintain the judgment of seeing and deciding rather than only the hand that draws.

Practices available to creators

The following are considerations derived by applying the studies above to creative work. None of them has been tested with creative practitioners, and the evidence is limited to analogies from findings in other fields.

  1. Generate the first ideas without AI. At the beginning of the divergent stage, produce one’s own ideas first and then place AI ideas beside them. Evidence: an experiment in which AI images strengthened fixation and narrowed the range of ideas (Wadinambiarachchi et al. 2024), and experiments in which AI ideas made works more similar to one another (Doshi and Hauser 2024; Anderson et al. 2024). When one’s own ideas exist first, AI ideas are easier to treat as a point of comparison.
  2. Write evaluation criteria before seeing AI output. Before looking at the output, put into words what counts as good and what would count as a failure. Evidence: the argument that generative AI demands self-awareness of goals and well-adjusted confidence in evaluation (Tankelevitch et al. 2024), and the review concluding that automation bias cannot be prevented by attention alone (Parasuraman and Manzey 2010). Because deciding to “look carefully” does not prevent the bias, the criteria are externalized before looking.
  3. Return to manual work at the core of a piece rather than following fixed quotas. Instead of rules such as “draw one piece by hand every day,” stop the flow of AI output at decisions that concern the core of a piece, such as the concept, signature expression, and the depiction of other people or facts. Evidence: SAFO 17007 avoided quotas without scientific evidence and left the decision to the pilot in command (FAA 2017). The article also proposed reading only high-risk diffs by hand. Because what counts as the core differs between projects, the creator decides it in advance.
  4. Try arrangements in which AI serves as the monitor. The creator draws or writes, and the AI critiques the work. Evidence: Sellen and Horvitz recommended considering situations in which it makes more sense for AI to monitor humans (Sellen and Horvitz 2024). In this arrangement, the creator remains the PF, so the cycle between hand and judgment is less likely to be interrupted. However, the quality of AI critique and bias toward that critique have not been examined.
  5. Bring cases in which AI surprised the team back into critique and debriefs. Record projects in which an adopted AI idea later turned out to be off target and projects in which ideas converged after following AI suggestions, and discuss them in team critiques. Evidence: aviation training builds its items from what actually happened (the evidence-based training introduced in the article), and a meta-analysis found that debriefs improved effectiveness by approximately 25% on average (46 samples; Tannenbaum and Cerasoli 2013). As the article noted, making these sessions pass-or-fail tends to make reports more modest.
  6. Keep hands-on practice. Even as more work is delegated to AI, deliberately keep time for making things by hand. Evidence: the account that discernment develops through seeing the results of one’s own moves (Schön and Wiggins 1992), and the aviation premise that the PM is trained to perform as PF (FAA 2017; PARC/CAST 2013). The aim is not to be able to make everything without AI but to be able to produce a different direction by oneself when the flow of AI output is stopped.
  7. Know which directions the AI tends to favor. Just as mode awareness has been treated as a safety requirement in aviation (Sarter and Woods 1995), record, while using an AI system, which kinds of options it tends to favor, such as common compositions and common phrasings. The evidence is an analogy from the homogenization experiments (Doshi and Hauser 2024; Anderson et al. 2024), and whether individual creators can discern these tendencies has not been examined.

Where the aviation analogy does not reach

The article identified differences between aviation and software in the severity and reversibility of failures, the unit of monitoring, the type of skill, and the way abnormalities are noticed. In creative work, three of these differences become larger.

The first difference concerns whether a correct answer exists. Abnormalities in aviation are defined by instruments and rules, and errors in software can be checked against tests and specifications. Mediocrity in creative work has no definition, and the same option can be good or bad depending on the context. Therefore, training the capacity to take over centers on putting one’s own criteria into words and comparing them with the criteria of others.

The second difference concerns how failure becomes visible. Homogenization does not appear within a single work. It appears when the works of many creators are placed side by side. Individual practices alone may not prevent it, and mechanisms for viewing works side by side, such as critiques and competitions, may be needed.

The third difference concerns the type of skill. The article distinguished manual flight, in which motor skills carry a large weight, from reading code, which relies mainly on cognitive skills. In illustration and video, manual skills carry a large weight, and Bainbridge’s point about the loss of refinements in gain and timing may apply directly to part of the work. In contrast, judgments about concept and editing are cognitive skills, and according to the results of Casner et al., these may decline more readily. No study has measured which creative skills decline at what rate and how they recover.

Questions the research has not yet answered

No longitudinal study was found that measured, with tasks performed without AI, whether the judgment of creative practitioners (discernment and reframing) changes after continued use of generative AI. A design that compares performance without AI before and after the introduction of AI, as in the colonoscopy study (Budzyń et al. 2025), may also be possible with records of design critiques. None of the seven practices above has been tested. In particular, generating the first ideas without AI and assigning AI the role of monitor could be compared by adding one condition to the procedures of the fixation and homogenization experiments.

Unverified items

No unverified statements remain in the main claims. For Sarter and Woods 1995, Endsley and Kiris 1995, Parasuraman and Riley 1997, Ebbatson et al. 2010, Tannenbaum and Cerasoli 2013, Budzyń et al. 2025, Dell’Acqua et al. 2026, and Hou et al. 2025, the full text could not be obtained, and they were used only within the scope of their abstracts.

References

Author: Shuichiro Ogawa (Design Researcher / Consultant) About me →