Lesson 1: What Is Software Engineering¶

Previous: Overview | Next: Software Development Life Cycle

Software engineering is the disciplined application of engineering principles to the design, development, testing, deployment, and maintenance of software. It is more than writing code — it is a body of knowledge, a set of processes, and a professional practice aimed at producing software that is reliable, efficient, and maintainable at scale.

Difficulty: ⭐⭐

Prerequisites: - Some programming experience in any language - Familiarity with the basic idea of software projects

Learning Objectives: - Define software engineering and distinguish it from programming and computer science - Explain the origins and significance of the software crisis - Describe the essential characteristics of software that make it challenging to engineer - Identify the major categories of software systems - List the core principles that guide software engineering practice - Recognize the professional roles found in software engineering teams

1. Definitions and Scope¶

What Is Software Engineering?¶

The term "software engineering" was first used publicly at the 1968 NATO Science Committee conference in Garmisch, Germany, deliberately chosen to be provocative — to suggest that software development should be held to the same standards of rigor and discipline as other engineering fields.

Several authoritative definitions have been offered over the decades:

IEEE (Institute of Electrical and Electronics Engineers):

"The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software." — IEEE Standard 610.12-1990

Sommerville:

"Software engineering is an engineering discipline that is concerned with all aspects of software production from the early stages of system specification through to maintaining the system after it has gone into use."

Pressman:

"Software engineering encompasses a process, a collection of methods (practice), and an array of tools that allow professionals to build high-quality computer software."

What these definitions share is an emphasis on all aspects of production — not just coding, but specification, design, testing, maintenance, and the organizational and managerial concerns that surround them.

Software Engineering vs. Programming¶

Programming is a necessary component of software engineering, but the two are not synonymous.

Dimension	Programming	Software Engineering
Scope	Writing code to solve a problem	The entire lifecycle of software production
Time horizon	Hours to days	Months to years
Team size	Typically individual	Often 5 to 500+ people
Primary concern	Making it work	Making it work, maintainably, reliably, on schedule
Key artifacts	Source code	Code, specifications, designs, tests, plans, docs
Key skills	Language proficiency, algorithms	Process, communication, estimation, risk management
Success metric	Code runs correctly	Project delivered on time, budget, with required quality

A useful analogy: programming is to software engineering as bricklaying is to civil engineering. Bricklaying is an essential craft, but civil engineering also encompasses structural analysis, project management, materials science, regulatory compliance, and long-term maintenance planning.

Software Engineering vs. Computer Science¶

Computer science and software engineering are related but distinct disciplines:

Computer Science is primarily concerned with the theory underlying computation: algorithms, data structures, formal languages, complexity theory, logic, and the mathematical foundations of computing.
Software Engineering is primarily concerned with the practice of producing software systems: how to organize the development process, manage teams, ensure quality, and deliver products that meet user needs.

In practice, a software engineer draws on computer science theory but is ultimately evaluated on whether software gets built and delivered reliably.

2. The Software Crisis¶

Historical Context: NATO 1968¶

By the late 1960s, it became apparent that the software industry was in trouble. Large software projects were routinely:

Over budget: Costs far exceeded estimates
Late: Schedules slipped by months or years
Unreliable: Delivered systems contained serious defects
Unmaintainable: Code was so complex that adding new features was prohibitively difficult
Cancelled: Many projects were abandoned entirely after consuming significant resources

Notable examples from that era include the IBM OS/360 operating system project, which Fred Brooks documented in his 1975 book The Mythical Man-Month. OS/360 was massively over budget and late, despite employing thousands of programmers.

The 1968 NATO conference coined the term "software crisis" to describe this state of affairs. Edsger Dijkstra, Tony Hoare, and others argued that software development needed to become a true engineering discipline with formal methods, systematic processes, and measurable outcomes.

The Continuing Challenge¶

Decades later, many of the original problems persist. The Standish Group's CHAOS Report has tracked software project outcomes since 1994:

CHAOS Report (approximate historical averages):
+------------------+------------------+
| Outcome          | % of Projects    |
+------------------+------------------+
| Successful       | ~30%             |
| (on time, budget,|                  |
|  full features)  |                  |
+------------------+------------------+
| Challenged       | ~50%             |
| (late, over      |                  |
|  budget, reduced |                  |
|  features)       |                  |
+------------------+------------------+
| Failed           | ~20%             |
| (cancelled or    |                  |
|  never used)     |                  |
+------------------+------------------+

Software engineering as a discipline exists precisely to improve these statistics.

3. Why Software Is Hard: Brooks's Four Properties¶

In No Silver Bullet (1986), Fred Brooks identified four essential properties of software that make it inherently difficult to engineer, arguing that there is no single technique that will dramatically improve software productivity.

3.1 Complexity¶

Software systems contain more states than any other human artifact of comparable size. A simple program with 300 boolean variables has $2^{300}$ possible states — far more than the number of atoms in the observable universe. This complexity is essential, not accidental: it reflects the complexity of the real-world problems software must solve.

This essential complexity means: - You cannot fully test all possible states - Changes in one part of a system may have unexpected effects elsewhere - Understanding a large codebase takes significant time and effort

3.2 Conformity¶

Unlike physics, which has laws that software can rely on, software must conform to the arbitrary decisions of humans — regulatory requirements, business rules, hardware interfaces, legacy APIs, and organizational conventions. These constraints are often inconsistent, poorly documented, and subject to change.

Software cannot appeal to nature. It must conform to whatever the world demands, no matter how illogical.

3.3 Changeability¶

Software is expected to change constantly. Because software is perceived as "soft" (easy to change compared to hardware), stakeholders routinely demand modifications after delivery. Every successful software system is subject to pressure for change, and every change carries the risk of introducing defects.

This is in contrast to physical engineering artifacts: a bridge built to specification is not typically required to be redesigned to add a new lane next month.

3.4 Invisibility¶

Software has no physical form. Unlike a building (where you can see structural problems) or a mechanical device (where you can measure tolerances), software is invisible. Diagrams and documentation are imperfect representations. This makes it difficult to visualize the structure of a system, communicate its architecture, or spot problems before they manifest at runtime.

4. Types of Software¶

Software engineering methods must be adapted to the type of software being built. Major categories include:

4.1 Systems Software¶

Infrastructure-level software that provides services to other software. Examples: operating systems, compilers, device drivers, database engines, runtime environments.

Characteristics: performance-critical, close to hardware, long lifespans, high reliability requirements.

4.2 Application Software¶

Software that performs tasks directly for end users. Examples: word processors, spreadsheets, accounting systems, ERP systems.

Characteristics: large user bases, evolving requirements, usability as a primary concern.

4.3 Embedded Software¶

Software that controls hardware devices. Examples: firmware in medical devices, automotive control units, industrial machinery, consumer electronics.

Characteristics: strict resource constraints (memory, CPU), real-time requirements, safety-critical, difficult to update after deployment.

4.4 Web Software¶

Software delivered and accessed through web browsers. Examples: e-commerce platforms, social networks, web APIs, SaaS applications.

Characteristics: rapid change cycles, heterogeneous client environments, network and security concerns, scalability requirements.

4.5 Mobile Software¶

Applications for smartphones and tablets. Examples: iOS/Android apps, cross-platform mobile apps.

Characteristics: constrained resources, intermittent connectivity, platform fragmentation, frequent OS updates.

4.6 AI and Machine Learning Systems¶

Software whose behavior is learned from data rather than explicitly programmed. Examples: recommendation engines, image classifiers, language models, autonomous systems.

Characteristics: non-deterministic behavior, data dependencies, model versioning, explainability concerns, different testing paradigms.

5. Core Principles of Software Engineering¶

Over decades of practice and research, a set of enduring principles has emerged that guide good software engineering regardless of the specific methodology used.

5.1 Rigor and Formality¶

Software should be developed with sufficient rigor — precise specifications, systematic testing, documented designs. The appropriate level of formality depends on the domain: safety-critical systems (avionics, medical devices) warrant formal mathematical verification; a startup prototype may need only basic documentation.

5.2 Separation of Concerns¶

Divide a complex system into parts, each addressing a distinct concern. This principle manifests at every level: - Module level: Each module has a single, well-defined responsibility - Architectural level: Separate presentation, business logic, and data layers - Process level: Separate requirements from design, design from implementation

Separation of concerns reduces cognitive load and enables teams to work on different parts simultaneously.

5.3 Modularity¶

Organize software into discrete, independently developable, testable, and replaceable modules. Well-designed modules have: - High cohesion: Related functionality is grouped together - Low coupling: Modules depend on each other as little as possible

5.4 Abstraction¶

Expose only what is necessary; hide implementation details. Abstraction allows engineers to work at the appropriate level — a developer using a database driver does not need to understand disk block allocation algorithms.

Layers of abstraction are fundamental to managing complexity in large systems.

5.5 Anticipation of Change¶

Software will change. Good software engineering designs for change by: - Identifying likely sources of variation and isolating them behind interfaces - Avoiding hardcoded constants and magic numbers - Writing clear, documented code that others can understand and modify - Designing databases and APIs for extensibility

5.6 Generality¶

Where practical, prefer general solutions over special-case solutions. A general-purpose sorting function is more valuable than a function that sorts only one specific data structure.

However, premature generality can lead to over-engineering. The principle must be balanced against simplicity.

5.7 Incrementality¶

Build and deliver software in increments rather than attempting to complete the entire system before any part is released. Incremental delivery: - Provides early feedback from real users - Reduces the risk of building the wrong thing - Allows value to be delivered before the project is "done"

This principle underlies both iterative development models and modern agile methods.

6. Brief History of Software Engineering¶

Timeline of Software Engineering

1948-1960s  Early programming
            ├─ Machine code → assembly → high-level languages (FORTRAN 1957, COBOL 1959)
            ├─ Programs written by mathematicians and scientists
            └─ Software "crises" begin to appear as projects grow

1968        NATO Conference, Garmisch
            ├─ Term "software engineering" coined
            ├─ Software crisis recognized as a discipline-level problem
            └─ Call for systematic, engineered approach

1970s       Process models emerge
            ├─ Royce describes Waterfall model (1970)
            ├─ Structured programming (Dijkstra, Wirth)
            ├─ Jackson Structured Design, Yourdon-DeMarco SA/SD
            └─ Unix developed at Bell Labs

1980s       Maturation
            ├─ Boehm's Spiral model (1988)
            ├─ COCOMO estimation model
            ├─ SEI and CMM (Capability Maturity Model)
            ├─ IEEE software engineering standards
            └─ Object-oriented design (Booch, Rumbaugh, Jacobson)

1990s       Objects, patterns, and web
            ├─ UML standardized (1997)
            ├─ Design Patterns "Gang of Four" (1994)
            ├─ World Wide Web transforms software distribution
            ├─ Extreme Programming (XP) introduced (Beck, 1996)
            └─ CMMI released (2000)

2001        Agile Manifesto
            ├─ Scrum, XP, Kanban gain widespread adoption
            ├─ Lightweight processes replace heavyweight ones
            └─ Iterative, customer-centric development becomes mainstream

2010s       DevOps and cloud
            ├─ DevOps movement formalizes dev-ops collaboration
            ├─ Continuous integration/continuous delivery (CI/CD)
            ├─ Microservices architecture
            ├─ Infrastructure as code
            └─ Cloud platforms (AWS, GCP, Azure) change deployment model

2020s       AI-assisted engineering
            ├─ Large language models assist with code generation
            ├─ AI/ML systems raise new engineering challenges
            ├─ Platform engineering and developer experience (DevEx)
            └─ Software supply chain security as a discipline

7. Professional Roles in Software Engineering¶

Modern software engineering teams include a range of specialized roles:

Role	Primary Responsibilities
Software Engineer / Developer	Design, implement, test, and maintain code
Software Architect	Define system structure, major technology choices, cross-cutting concerns
Product Manager (PM)	Define what to build; own the product roadmap; represent business and user needs
Project Manager	Plan, schedule, and track project progress; manage risk and resources
QA Engineer / Test Engineer	Design and execute tests; maintain test infrastructure; report defects
DevOps / Platform Engineer	Build and maintain CI/CD pipelines, infrastructure, and deployment systems
Site Reliability Engineer (SRE)	Ensure system reliability and availability in production; define SLOs
Technical Writer	Create and maintain documentation for developers and end users
Business Analyst	Bridge business needs and technical implementation; write requirements
UX Designer	Design user interfaces and user experiences
Security Engineer	Identify and mitigate security vulnerabilities; define secure coding standards
Data Engineer	Build data pipelines and infrastructure; manage data quality

In small organizations, one person may wear multiple hats. In large organizations, these roles are more clearly delineated. Understanding all of these roles — even if you only occupy one — makes you a more effective collaborator.

8. Summary¶

Software engineering is the application of systematic, disciplined, and quantifiable approaches to software development. It emerged as a response to the "software crisis" of the 1960s, when large software projects routinely failed to deliver on time, on budget, or with sufficient quality.

Key takeaways: - Software engineering is broader than programming: it encompasses the entire lifecycle and all organizational concerns - Software is inherently complex due to its complexity, conformity, changeability, and invisibility (Brooks) - Core principles — separation of concerns, modularity, abstraction, incrementality — apply across all methodologies - Different types of software (embedded, web, AI) require different engineering approaches - The field has evolved continuously from the 1960s through today's AI-assisted development era

Practice Exercises¶

Exercise 1: Compare and contrast software engineering with civil engineering. Identify three similarities and three fundamental differences. Which of Brooks's four properties has no direct analog in civil engineering?

Exercise 2: You are a sole developer maintaining a personal script that automates your email sorting. Identify which software engineering principles you should still apply, and which are less critical at this scale. Justify your reasoning.

Exercise 3: Look up one well-known software project failure (examples: HealthCare.gov 2013 launch, Ariane 5 Flight 501, Knight Capital incident 2012). Write a one-page analysis: What went wrong? Which software engineering principles were violated? What could have been done differently?

Exercise 4: Interview (or research online) someone who works in a software engineering role you are unfamiliar with (e.g., QA engineer, technical writer, SRE). Describe their day-to-day responsibilities and how their work intersects with software developers.

Exercise 5: Brooks claimed in 1986 that there is "no silver bullet" — no single development, technique, or management practice that will produce a tenfold improvement in productivity. Review arguments for and against this claim in light of modern developments (AI code assistants, cloud infrastructure, mature frameworks). Do you agree or disagree? Provide evidence.