Shuichiro Ogawa
日本語

Notes · updated 2026-08-07

Overview

This note organizes the industry reality of validating system architecture through simulation, emulation, and rehearsal before operating real infrastructure or services, and of learning through those means. It covers four areas: (a) pre-production validation techniques (deterministic simulation testing, chaos engineering, local emulation, industrial use of formal methods, digital twins), (b) exercise-based learning services (sandbox labs, cyber ranges), (c) standards and public guidance, and (d) skill demand. Sources were collected across four streams — T1 (public agencies and standards), T2 (independent surveys), T3 (first-person testimony from builders and operators) — plus labor-market data, yielding 46 data points (1 heavy exclusion plus a batch of methodology-free reports excluded, 2 marketing claims separated). The scholarly lineage (learning sciences of simulation-based education, formal-methods research literature) is out of scope here and will be collected separately in academic mode.

Public Agencies and Standards (T1)

The shared vocabulary of pre-production validation is being standardized along two lines: digital twins and distributed simulation.

ISO/IEC 30173:2023 defines a digital twin as a fit-for-purpose digital representation of an observable entity with synchronization between the entity and its representation. The synchronization requirement conceptually distinguishes it from one-way simulation. NIST IR 8356 (2025), by contrast, positions digital twins as an extension of existing modeling-and-simulation capabilities, so the definitions oscillate between continuity and distinction. In manufacturing, the ISO 23247 series (2021) specifies a reference architecture that anticipates simulation-based pre-validation. On the economic side, NIST AMS 100-61 (2024) estimates the potential impact of digital-twin adoption in U.S. manufacturing at $37.9B in total, conditional on data-tracking investment costs at or above the 85th percentile (a conditional upper-bound estimate, not a measured market size).

In the distributed-simulation lineage, IEEE 1516-2025 (HLA 4) was published in 2025 as a major revision covering cloud and container support, and IEEE 1730-2022 (DSEEP) defines the design-to-execution process framework. Architecture description itself is standardized by ISO/IEC/IEEE 42010:2022.

On the learning-services side, the glossary derived from NIST SP 800-50 Rev.1 (2024) defines a cyber range as a web-based environment that provides realistic training, scenarios, challenges, and exercises in a safe sandbox environment. For organization-scale rehearsal, ENISA published an end-to-end exercise methodology (2026) and keeps official records of the pan-European Cyber Europe exercises (after-action reports, 2022 and 2024). Much of the full text of these standards and guides sits behind paywalls or in PDFs, so clause-level verification of definitions remains open (recorded as “[要一次検証]” (primary verification needed) in the corpus).

Facts from Independent Surveys (T2, position talk removed)

This section carries only factual portions, with recommendations and future assertions stripped. Two Gartner-derived figures could not be methodologically verified behind the paywall and are retained with verification flags.

On the shape of pre-production environments, the ThoughtWorks Technology Radar shows a consistent shift. Vol.31 (2024) placed enterprise-wide integration test environments on Hold, arguing they invariably become a bottleneck and provide a false sense of security due to environment discrepancies, recommending ephemeral environments and contract testing instead. It simultaneously placed in-memory component testing on Adopt, and Vol.32 (2025) promoted fuzz testing to Adopt. The practical recommendation is thus moving toward isolated, reproducible pre-validation rather than maintaining shared production-like environments. Chaos engineering itself last appeared on the Radar as Trial in 2019; its absence since is explicitly attributed to limited Radar real estate, not to a downgrade.

On adoption rates, the CNCF Annual Survey 2024 (750 respondents, self-selection bias) reports 28% of organizations with a dedicated platform engineering team and 41% with a multi-team approach. Production use of chaos engineering stood at roughly 12% in the CNCF 2023 survey, with about 40% evaluating. A Gartner Peer Community survey reports 59% adopting chaos engineering, but its sample and definitions could not be verified. Because the gap between 12% and 59% can stem from different populations (cloud-native practitioners vs. Gartner clients) and different definitions of “adoption,” neither figure can be quoted as a single adoption rate. DORA 2024 reports a negative association between AI adoption and delivery performance (a 25% increase in AI adoption associated with -7.2% delivery stability; run by Google Cloud, hence judged partial), which reads as evidence that pre-production validation may gain value in the generative-AI era.

Testimony from Builders and Operators (T3)

First-person testimony traces three lineages of pre-production validation.

The deterministic simulation testing (DST) lineage begins with FoundationDB. In his 2014 talk, Will Wilson described replacing network, disk, and machines with a software layer so that failures become fully controllable and reproducible, re-running with the same seed to detect any leakage of nondeterminism. At Antithesis, which he co-founded, Wilson says the company sacrificed the purity of FoundationDB’s single-process discipline and opened deterministic execution to existing codebases via a hypervisor, noting that customers start with nightly runs rather than pre-merge checks (a speed-versus-coverage tradeoff). TigerBeetle split its simulator into Safety mode (serializability under random faults) and Liveness mode (availability whenever a quorum exists), and reports discovering a “resonance bug” that restarts had been masking. Antithesis’s own documentation lists constraints: determinism cannot be maintained without mocking external dependencies, and retrofitting legacy systems is hard (superiority claims were separated as marketing claims).

For formal methods in industry, AWS engineers reported first-hand in 2015 that at S3 scale they applied TLA+ to design problems intractable to conventional testing, with seven teams adopting it and all finding value. The successor P language, per designer Ankush Desai, has been adopted by more than 80 service teams and 600 developers, catching critical bugs before implementation in efforts such as S3’s strong-consistency migration.

Chaos engineering and organizational rehearsal turn the real environment itself into a learning apparatus. Netflix’s Simian Army (2011) assumed that failures are inevitable and deliberately triggered them during business hours so engineers would be present to build response capability. Chaos Kong (2015) rehearsed region-wide failure, and the team cites its response to an actual DynamoDB outage days later as evidence of effect. Kripa Krishnan, who led Google’s DiRT for about nine years, described detecting not only technical weaknesses but operational single points of failure by deliberately excluding specific experts from exercises.

Testimony on local emulation and learning services exposes the fidelity-versus-operability tradeoff. LocalStack’s co-founder describes concentrating on 15-20 core services rather than replicating 400+ AWS services, and skipping AWS’s intentional latencies in favor of developer productivity. Among exercise-based learning services, the founder of iximiuz Labs details Firecracker microVM constraints (no nested virtualization, custom kernels required) and a scale ceiling of 100-200 concurrent playgrounds without redesign, while SadServers’ founder cites the absence of “real server environments with a specific problem and a win condition” (after Katacoda’s 2022 shutdown) and the standardization of hiring interviews as motives. Learning services sell constrained environments with well-posed problems, not faithful reproductions of real infrastructure.

Labor Market and Skill Demand

The market for roles carrying these validation and reliability skills is hard to isolate statistically. SRE has no dedicated BLS occupational code; it is dispersed across several occupations with no aggregate figure. For adjacent occupations, Computer Network Architects show a median wage of $130,390 with a +12% outlook (2024-2034), and the Software Developers group $133,080 with +15% (BLS; primary pages unreachable, flagged for verification). On the demand side, U.S. tech job postings were down 36% from February 2020 as of July 2025, while data-center-related postings more than tripled between 2023 and 2026 (Indeed Hiring Lab; no SRE-specific breakdown). Self-reported salary data differ by definition — Glassdoor (base-centric, average $172,508) versus Levels.fyi (median total compensation $205,000) — and cannot be compared directly. On learning methods, Stack Overflow 2024 (65,437 responses) reports online resources at 82.1% and e-courses/certifications at 50%, but has no question isolating exercise- or simulation-based learning.

Applying This to Student Education

The background of this survey is the question of how to teach system design to students. Two parts of the industry data bear directly on selecting teaching material: the design philosophy of exercise-based learning services, and pre-production validation material that runs entirely in a browser.

Exercise-based learning services sell well-posed problems with win conditions, not faithful reproductions of real infrastructure. iximiuz Labs offers browser-accessible microVM playgrounds for container and Kubernetes exercises, and SadServers offers scenario drills on “real server environments with a specific problem and a win condition” for Linux troubleshooting. Translated into course design, the question an institution should ask is not “how closely can we replicate production” but the quality of problem posing: what students diagnose under constraints, and what counts as solved. The founders’ published constraints (no nested virtualization, custom kernels required, a ceiling of 100-200 concurrent playgrounds) also help estimate the cost and infrastructure requirements of building an equivalent environment in-house.

On the validation-technique side, browser-only material already exists. TigerBeetle publishes its distributed-database simulator in the browser, letting an observer watch consensus behavior while data corruption is injected at 8% of reads and 9% of writes (3.3 seconds of simulation corresponding to 39 minutes of real time). The idea of deterministic simulation testing (fully controlling failures in a software layer and re-running with the same seed to detect nondeterminism) translates into an assignment that asks students to enumerate possible failures and define, in language, the properties that must hold under them. The AWS experience with TLA+ and the P language demonstrates, from the industry side, that critical bugs can be found at the design stage before implementation, which motivates bringing formal methods into design education.

The organizational-rehearsal lineage is suggestive for team-based coursework because it validates people and procedures rather than technology. Google DiRT’s practice of deliberately excluding specific experts from exercises is a device for detecting operational single points of failure in personalized knowledge, and can be imitated directly as a role-play in which a group member is temporarily removed from an assignment.

Module-Composition Learning Services (Addendum)

The addendum question is whether services exist that teach the consequences of architecture choices through composing modules — for example, that omitting a Load Balancer makes a service unusable under load, or that misconfiguring a VPC makes published resources unreachable. Two additional collection streams were run (vendor-primary and developer-voice; the ledger is in the corpus addendum).

The closest matches to this shape are not the cloud vendors’ official material but free simulators and games by individual developers. Server Survival is a free browser game in which the player places 26 component types (GeoDNS, CDN, load balancer, cache, databases, message queues) against growing traffic and watches failures (over capacity, no route) through real-time RPS, error-rate, and latency displays. The open-source system-design-simulator wires 35 component types freely and pushes 1K-500K requests per second through the design, detecting bottlenecks and cascading failures. Paperdraw adds chaos switches (traffic spike, cache-miss storm, network partition, component crash) to a wired design, and systemdesignsimulator.org’s stress-test mode shows which component saturates first. All are free tools by individual developers, so model fidelity and maintenance continuity are not guaranteed — a caveat for adoption as course material.

The cloud vendors’ official exercises center on building the correct configuration; consequence-experiencing exercises exist only around their fault-injection services. AWS Cloud Quest and SimuLearn, Google Cloud Skills Boost, and Microsoft Learn labs all have learners build VPCs and load balancers in real cloud environments with automated grading, but no exercise observing the consequences of a missing LB or a misconfigured VPC could be confirmed at any of the three vendors. The closest official material is the AWS Well-Architected Labs Reliability 300 lab (injecting EC2, RDS, and AZ failures with the Fault Injection Service and observing behavior) and the official FIS tutorial (measuring an EC2 instance losing connectivity to S3 — effectively the VPC-unreachability experience). Azure Chaos Studio offers comparable managed fault injection. All of these run on real cloud accounts, so deploying them to students requires cost and permission guardrails.

Between the two sit repair-style exercises and zero-cost breakable environments. SadServers has time-boxed scenarios for fixing broken HAProxy and Traefik round-robin configurations (Tarifa, San Juan) and diagnosing unreachability (Batumi), teaching LB misconfiguration consequences from the repair side. LocalStack lets students compose VPCs, subnets, and ALBs on a local emulator, so misconfiguration failures can be repeated at zero cost (students verified through the GitHub Student Developer Pack get a paid plan for free). Consequence visualization is not automatic, though; learners read the logs themselves. For single components, Sam Rose’s interactive Load Balancing and Queueing visualizations serve as introductory material for experiencing algorithm differences and queue-overflow drops hands-on. The complementary side — building a correct design and receiving critique on what is missing, rather than experiencing breakage — is organized separately in Tools for Learning System Design.

What this survey cannot say, however, is anything about educational effectiveness. No independent survey on the industry side shows whether exercise- or simulation-based learning outperforms lectures (Stack Overflow 2024 has no such question), and no usage survey of system-design exercise services was found. If evidence of effectiveness is needed for adoption decisions, peer-reviewed literature from the learning sciences and educational technology must be collected separately in academic mode.

Gaps

  • No independent quantitative survey exists on the adoption of deterministic simulation testing; current evidence is limited to builder testimony (T3).
  • The adoption-rate gap for chaos engineering (12% vs. 59%) remains unresolved; an independent survey with aligned populations and definitions is missing.
  • No methodologically explicit survey was found on the usage of commercial exercise-based learning services beyond cyber ranges.
  • Major consultancies (McKinsey, BCG, Deloitte) offer no methodology-explicit quantitative work specific to this topic; T2 relies on technology radars and community surveys.
  • Effectiveness research on simulation-based learning belongs to the scholarly side (learning sciences, educational technology) and will be collected separately in academic mode.
  • Module-composition simulators (see the addendum) are concentrated in free tools by individual developers; no survey exists on institutional adoption, model fidelity, or maintenance continuity. No official vendor exercise comparing load consequences with and without a load balancer could be confirmed at any of the three vendors.

References

All accessed 2026-08-07. For ledger details (methodology, position judgments, all verification flags), see source/review/system-architecture-simulation-learning/industry.md.

Public agencies and standards (T1)

Independent surveys (T2)

Builder and operator testimony (T3)

Addendum (module-composition learning services)

Labor market

← All Notes · Home