The Laser Weapon System (LaWS) aboard USS Ponce. This is what started the rabbit hole. Photo: U.S. Navy.

My first reaction: how does light do that?

Light is the thing that comes out of a lamp. It’s what reads barcodes at the grocery store. My cat chases a laser pointer dot around the living room. How do you get from that to burning a military drone out of the sky?

And then the obvious follow-up: is it even the same thing? Is the physics behind a three-dollar laser pointer and a naval weapons system actually the same physics? Or are they just both called “laser” the way a toy car and a Formula 1 car are both called “car”?

Turns out: it really is the same physics. Every laser ever built (the pointer, the barcode scanner, the eye surgery machine, the drone-killer) works because of one quantum mechanical trick: a photon hits an excited atom, and two identical photons come out. Same wavelength, same direction, perfectly synchronized. The photon clones itself.

Einstein predicted this would happen in 1917 while he was trying to make some equations balance. He called it “stimulated emission,” thought it was a curious theoretical detail, and moved on. Nobody figured out what to do with it for 43 years.

I went down the rabbit hole on this, and it goes deep. The materials that make lasers possible (ruby crystals, rare earth elements, certain gas mixtures) have these properties basically by accident. People discovered them because they noticed strange glows under certain conditions and got curious enough to investigate. The whole field exists because some physicists looked at a faintly glowing crystal and thought huh, that’s weird instead of walking past it.

This post starts with a light bulb and ends with lasers that briefly outpower every power plant on Earth combined. It’s a long one. Let’s go.

Part 1: Let’s Start With a Light Bulb

Photo: Joy Singh / Pexels

You flip a switch. A light bulb turns on. Simple, right? But what’s actually happening inside that bulb is your first step into understanding why lasers are so weirdly special.

Inside an old-school incandescent light bulb, there’s a tungsten wire. When you flip the switch, electricity flows through that wire, and because tungsten has electrical resistance, the wire heats up. A lot. We’re talking about 2,500°C (4,500°F) kind of hot. And when things get that hot, something interesting happens at the atomic level.

The Electron Shuffle: A Chemistry Refresher

Remember from chemistry class that electrons orbit atoms in specific energy levels? Think of these like the floors of a hotel. An electron can be on the ground floor, the second floor, the third floor, etc., but it can’t just hang out in the stairwell between floors. Those in-between spaces? Not allowed. Quantum mechanics says no.

Now, when you heat up tungsten to ridiculous temperatures, you’re basically pumping energy into the atoms. This energy causes electrons to get excited, and yes, that’s literally the physics term. An excited electron absorbs energy and jumps from a lower energy level to a higher one. It’s like someone giving our electron enough energy to take the elevator up several floors.

But here’s the thing: electrons are homebodies. They don’t want to be on the higher floors. The ground state, the lowest energy level, is where they’re most stable and most comfortable. So after a tiny, tiny fraction of a second (sometimes nanoseconds, sometimes microseconds) they fall back down to a lower energy level.

And here’s the crucial part: when an electron drops from a higher energy level to a lower one, it has to release the energy it gained. Energy can’t just disappear. And it releases that energy as a photon, a particle of light.

The energy difference between the two levels determines the photon’s wavelength, which we perceive as its color. Big energy difference means a high energy photon, which corresponds to blue or ultraviolet light. Small energy difference means a low energy photon, which gives you red or infrared light. The relationship is exact and mathematical: the photon’s energy equals Planck’s constant times its frequency, and frequency is related to wavelength by the speed of light. This is why each type of atom has its own characteristic spectrum, its own fingerprint of colors it can emit.

Why Regular Light is Kind of Messy

So in your light bulb, you’ve got trillions upon trillions of tungsten atoms, all at different temperatures, with electrons constantly jumping up and falling back down. But here’s the critical insight: this process is completely chaotic and random.

Electrons are jumping to different energy levels randomly because the thermal energy doesn’t care which level an electron ends up in. It’s just chaos, heat being absorbed however it can be. They’re falling back down at random times because spontaneous emission is, well, spontaneous. There’s no trigger, no cause. An excited electron just has some probability of falling down each moment, and eventually it does. The photons are shooting out in completely random directions because there’s nothing to favor one direction over another. They go up, down, left, right, forward, backward, all 360 degrees worth of possibilities. And the photons are all different wavelengths because electrons are dropping between all sorts of different energy levels, not just one specific transition.

The result? Light that spreads out in all directions, with a broad spectrum of colors, which is why incandescent light looks yellowish-white rather than a pure color, and where all the light waves are completely out of sync with each other. The peaks of one wave don’t line up with the peaks of another. Each photon is doing its own thing, like a crowd of people all talking at once instead of singing in unison.

This type of light is called incoherent light. It’s messy, it’s chaotic, it’s random. And it’s perfectly fine for lighting up your room. The incoherence doesn’t matter when you just want to see where you’re walking.

But what if we wanted something different? What if we wanted all the photons to be the same wavelength, to travel in the exact same direction, and to be synchronized with all their waves lined up perfectly? That’s where lasers come in.

Part 2: What Makes Lasers Special? (And Why Should We Care?)

Before we dive into how lasers work, let’s understand why anyone wanted to build one in the first place. What properties of laser light make them so useful that people spent years trying to figure out how to create them?

The Four Key Properties

Laser light has four special characteristics that regular light doesn’t have, and each one enables different applications.

First, there’s monochromaticity, which means the laser is essentially one pure color. Not “reddish” or “bluish” but a single, precise wavelength. A sodium vapor street lamp might look yellow and seem like one color, but it’s actually emitting several different wavelengths in the yellow region. A laser, on the other hand, might emit light at exactly 632.8 nanometers, and nothing else. This precision is useful in surprising ways. If you’re measuring distances with light (which is what surveyors do, what satellites use to map the Earth’s surface, what scientists use in countless experiments) having exactly one wavelength means your measurements can be incredibly precise. You can calculate distances based on counting the number of wavelengths that fit in that distance, and if your wavelength is fuzzy, your measurement is fuzzy. In fiber optic communication, different wavelengths can carry different signals down the same fiber, but only if each laser is exactly one wavelength with no spread. And in medical applications, different tissues absorb different wavelengths differently, so you can use a laser to target specific structures. A laser tuned to be absorbed by blood vessels but not surrounding tissue can treat vascular problems without damaging healthy tissue.

Second is coherence, which means all the light waves are synchronized. Their peaks and troughs line up. This is like the difference between a stadium of people clapping randomly versus everyone clapping in perfect unison. The unified clapping is way more powerful and you can hear it much farther away. When light waves are coherent, they reinforce each other constructively. Where two peaks meet, they add up to an even bigger peak. This means the light doesn’t spread out and dissipate as quickly. Coherent light can travel much farther while maintaining its intensity. This coherence also enables interference patterns. When you combine two coherent light beams, they create interference fringes, areas where they add up constructively and areas where they cancel out destructively. This is the basis for holography, where you can record three-dimensional images, and interferometry, where you can measure tiny distances or detect gravitational waves by looking at how interference patterns shift.

Third is directionality. A laser beam stays tightly focused and travels in essentially one direction. A regular light bulb radiates light in all directions equally, it’s isotropic. Even if you put a reflector behind it to create a spotlight, most of the light still spreads out in a cone. A laser pointer, on the other hand, creates a beam that barely spreads at all. You can see the tiny dot on a wall across a room, and the beam is nearly the same width at the wall as it was when it left the pointer. This happens because all the photons are traveling in the same direction, parallel to each other. Actually, they’re not perfectly parallel, there’s always some small divergence, but it’s remarkably small compared to any other light source. This directionality is what makes laser pointers work. It’s why you can aim lasers at things and hit exactly what you’re aiming at. It’s why laser cutting works, you can focus all that energy onto a tiny spot. And it’s why you can send laser signals long distances through fiber optics or even through space without losing the signal.

Fourth is high intensity. Because lasers concentrate all their energy into one wavelength, traveling in one direction, in sync, you can achieve incredibly high power densities. Intensity isn’t just about total power, it’s about power per unit area. A 1-watt laser can cut through materials that a 100-watt light bulb couldn’t touch, because all that laser energy is concentrated into a spot that might be a fraction of a millimeter across, while the light bulb’s energy is spread over an entire room. If you focus a laser beam through a lens, you can achieve power densities high enough to vaporize almost any material. This is why laser cutting works. This is why you can use lasers for welding, for drilling tiny holes in materials, for surgery where you need to precisely remove tissue.

So now that we know what we’re trying to achieve (monochromatic, coherent, directional, intense light) the question becomes: how the heck do we make light do these weird things?

Part 3: The Quantum Mechanical Magic (The Physics Foundation)

To understand how lasers work, we need to go deeper into what happens when light interacts with atoms. There are three key processes, and understanding them is the secret to understanding everything about lasers. This is the stuff Einstein figured out in 1917.

The Three Sacred Processes

The first process is absorption. An atom is sitting there with its electron in a low energy state, just minding its own business. A photon comes along, traveling through space. Now, here’s the thing: the photon can’t just randomly interact with the atom. The photon has to have exactly the right amount of energy. If the photon’s energy matches the energy difference between the electron’s current level and some higher level, then the electron can absorb that photon and jump up to that higher energy level. The photon disappears (its energy is now stored in the excited electron) and the electron is now in a higher energy state. It’s like the electron used the photon as currency to buy a ticket to a higher floor.

If the photon’s energy doesn’t match any allowed transition, it just passes by without interacting. This is why glass is transparent to visible light, the energy levels in silicon dioxide atoms are spaced such that visible photons don’t have the right energy to cause transitions. The photons just pass through.

The second process is spontaneous emission. The excited electron is unstable up there on that higher energy level. It wants to fall back down. After some random amount of time (could be nanoseconds, could be microseconds, could be milliseconds depending on the specific transition and atom) it spontaneously falls back down to a lower energy level. When it does, it has to release the energy it’s been storing, and it releases it as a photon. That photon has energy equal to the difference between the two levels.

But here’s the key thing: this happens randomly. There’s no trigger, no external cause. It’s like radioactive decay, you can’t predict exactly when a specific excited electron will emit, you can only know the probability. And when it does emit, the direction is random. Up, down, sideways, any direction is equally likely. This is what’s happening in your light bulb. This is what’s happening in a neon sign, in a fluorescent tube, in any normal light source. Spontaneous emission gives you that chaotic, incoherent light.

The third process is stimulated emission, and this is where Einstein’s genius comes in. Imagine an electron is sitting in an excited state, just hanging out, waiting to eventually fall back down via spontaneous emission. Now imagine another photon comes along, a photon that has exactly the same energy as the photon that would be released if the electron dropped down.

Einstein’s mathematics showed that this incoming photon can actually trigger the electron to drop down immediately, before it would have spontaneously emitted. And here’s the absolutely crucial part: when the electron drops down via stimulated emission, it releases a new photon that is identical to the triggering photon in every way. Same wavelength? Yes. Same direction? Yes. Same phase, meaning its wave peaks and troughs line up with the original photon? Yes.

So one photon goes in, and two identical photons come out, both traveling in the same direction, perfectly synchronized. The photon cloned itself.

This is the entire secret of how lasers work. This is the process Einstein predicted. This is what we’re going to exploit.

The Problem: Population Inversion

But here’s the thing: under normal circumstances, stimulated emission basically never happens. Why not?

Because most atoms, most of the time, have their electrons in the ground state. They’re not excited. So when a photon comes along, it’s much more likely to be absorbed by an electron jumping up than to cause stimulated emission from an already-excited electron.

Let’s say you have a million atoms, and you send a beam of light through them. If all the electrons are in the ground state, photons will be absorbed, electrons will be excited, and then those electrons will spontaneously emit in random directions. You don’t get amplification. You get absorption and then random re-emission.

For stimulated emission to dominate, you need more atoms with electrons in the excited state than in the ground state. This weird, unnatural condition is called population inversion. It’s called that because normally the lower state is more populated, but we need to invert that.

Population inversion doesn’t occur naturally. Left to themselves, atoms always have more electrons in lower states than higher states. That’s just thermodynamics, systems naturally settle into lower energy states.

To make a laser, you have to force population inversion to happen. You have to pump energy into the system faster than the excited electrons can fall back down. You have to fight against the natural tendency toward equilibrium.

This is the fundamental challenge of building a laser: creating and maintaining population inversion.

Part 4: The First Laser (The Ruby Laser Story: High Level)

Now that we understand the physics principles, let’s see how Theodore Maiman actually built the first laser in 1960. We’ll start with the high-level view, and then dive deeper into why his approach worked.

Maiman was working at Hughes Research Laboratories in California. Other researchers were trying to build lasers using gases, because the theory suggested gases might work better. But Maiman had a hunch about ruby. He knew ruby had the right kind of energy levels, and ruby crystals could be grown large and of high quality.

The Apparatus

A ruby laser. The pink rod is the ruby crystal, surrounded by a coiled xenon flash lamp. Image: Wikimedia Commons, Public Domain.

Maiman’s laser was elegantly simple, at least in concept. He had a ruby rod about the size of a finger, roughly 1 cm in diameter and a few centimeters long. Ruby is aluminum oxide (Al₂O₃) with a small amount of chromium atoms mixed in, which gives it that characteristic red color. Both ends of the ruby rod were polished flat and parallel, then coated with silver to make them reflective. One end was coated to be about 100% reflective, the other about 95% reflective, letting 5% of the light through.

Wrapped around the ruby rod was a helical xenon flash lamp, essentially a very bright, intense camera flash that could produce a brilliant burst of white light for a few milliseconds. The flash lamp was coiled in a spiral around the ruby, so it could shine light into the ruby from all sides.

The whole thing was small enough to hold in your hands.

How It Actually Works: The Step-by-Step Dance

When Maiman fired the flash lamp, here’s what happened:

The flash lamp produces a brilliant burst of white light, containing many wavelengths. This light shines into the ruby from all sides. The chromium atoms in the ruby absorb certain wavelengths, specifically the blue and green parts of the spectrum. Electrons in the chromium atoms jump from low energy states up to high energy states. This happens for billions and billions of chromium atoms throughout the ruby.

These excited electrons then quickly drop down to a special intermediate energy level and stay there for a relatively long time, milliseconds, which in atomic terms is an eternity. This accumulation of excited atoms is what creates population inversion.

Now the cascade begins. Eventually, one of those excited electrons spontaneously emits. It drops down to a lower energy state, releasing a red photon at 694.3 nanometers. This photon travels through the ruby in some random direction.

Most of these spontaneous photons are traveling at angles and just exit through the sides of the ruby rod. They’re lost, contributing nothing to the laser beam. But occasionally, just by chance, a spontaneous photon happens to be traveling along the axis of the ruby rod, parallel to the length.

This photon travels through the ruby, and as it does, it passes by other chromium atoms that have electrons in excited states. Each time it encounters such an atom, there’s a chance it triggers stimulated emission. When it does, now there are two photons, both traveling in the same direction, both perfectly in phase.

These two photons continue traveling through the ruby, potentially triggering more stimulated emission. Two becomes four, four becomes eight, eight becomes sixteen. The number of photons grows exponentially.

Eventually, these photons reach the mirror at the end of the ruby rod. The mirror reflects them back through the ruby in the opposite direction. As they travel back, they trigger even more stimulated emission. The number of photons builds up even more.

The photons bounce back and forth between the mirrors, and with each pass through the ruby, they trigger more and more stimulated emission. The light intensity builds up inside the optical cavity formed by the two mirrors.

But remember, one mirror is only 95% reflective. Each time the light hits this partially reflective mirror, 5% of it leaks through. At first, this leakage is negligible. But as the intensity builds up inside the cavity, that 5% leakage becomes substantial. That’s your laser beam.

The laser beam that emerges is monochromatic (694.3 nm red light), directional (traveling along the axis of the rod), coherent (all triggered by stimulated emission from photons that were already in phase), and intense (amplified by many passes through the ruby).

The Numbers Are Insane

The statistics behind what’s happening are genuinely absurd. When an electron spontaneously emits a photon, it can go in any direction. If we think about it three-dimensionally, there are all possible directions in 3D space, technically 4π steradians of solid angle. For a photon to travel along the axis of the ruby rod and enter the optical cavity mode, it needs to be within a very small solid angle, determined by the diameter of the rod and the length.

For a typical ruby laser, maybe only 1 in 10,000 photons that spontaneously emit happen to be traveling in the right direction to contribute to the laser beam. The other 9,999 out of 10,000 just fly out the sides of the ruby and are lost.

Think about this: it’s like trying to fill a swimming pool by throwing water balloons in random directions and hoping one lands in the pool. It seems impossibly wasteful.

But here’s why it works anyway: once you have even a few photons traveling in the right direction and bouncing between the mirrors, stimulated emission takes over. Stimulated emission, unlike spontaneous emission, is directional. The new photon travels in the same direction as the triggering photon. This is the magic trick.

So you start with maybe just 1 photon going in the right direction. Through stimulated emission, that becomes 2, then 4, then 8, then 16. The growth is exponential. Even though you start with very few photons going the right way, they multiply so quickly that soon they completely dominate over the spontaneous emission happening in random directions.

The gain per pass through the ruby is important. If you have strong population inversion, one photon might trigger enough stimulated emission to create 1.5 photons per centimeter it travels. After traveling through a few centimeters of ruby, you’ve doubled your number of photons. After the photons bounce back and forth ten times, you’ve multiplied your initial photon count by an enormous factor.

The optical cavity is essential for this. Without the mirrors, the photons would just pass through once and exit. They wouldn’t have multiple chances to trigger stimulated emission. With the mirrors, the photons can bounce back and forth hundreds or thousands of times before they finally leak out through the partially reflective mirror, and each pass gives them opportunities to multiply.

So yes, the initial probability is low. Yes, most spontaneously emitted photons are wasted. But exponential growth is extraordinarily powerful, and the optical cavity gives the photons time to multiply into a torrent of synchronized light.

Part 5: Why Can Some Materials Do This? (The Secret of Laser Media)

Now that we’ve seen how a laser works at a high level, a natural question arises: why does ruby work as a laser medium? Why does a helium-neon gas mixture work? Why does a CO₂ molecule work? What makes certain materials special, while most materials can’t be used to make lasers at all?

The answer lies in the structure of their energy levels, and specifically in something called metastable states.

The Metastable State: Nature’s Battery

Remember, for a laser to work, you need population inversion. You need lots of atoms with electrons in an excited state. But normally, excited electrons fall back down really quickly via spontaneous emission, sometimes in nanoseconds. If excited electrons only stayed excited for nanoseconds, you’d need to pump energy into the system absurdly fast just to keep up with the electrons falling back down. You’d be like someone trying to fill a bathtub with the drain wide open.

This is where metastable states save the day. A metastable state is an excited energy level where an electron can stay for a relatively long time before falling back down. “Relatively long” in quantum mechanics might mean milliseconds instead of nanoseconds, but that’s a million times longer. It’s like the difference between water draining from a bathtub in one second versus 11 days. Suddenly, you can fill the bathtub faster than it drains.

Why Do Metastable States Exist?

This is where quantum mechanics gets interesting. Not all transitions between energy levels are equally likely. Some transitions happen very quickly, and others happen very slowly. This has to do with quantum mechanical selection rules.

When an electron emits a photon and drops to a lower energy level, the most common mechanism is electric dipole radiation. For this to happen efficiently, certain quantum mechanical conditions need to be met. The electron’s orbital angular momentum needs to change by exactly one unit (ΔL = ±1), and there are rules about the magnetic quantum number and spin as well.

If a transition satisfies all these selection rules, it’s called an “allowed” transition, and it happens quickly, nanoseconds or faster. If a transition violates these selection rules, it’s called a “forbidden” transition. Now, “forbidden” doesn’t mean impossible, quantum mechanics is probabilistic, not deterministic, but it means the probability per unit time is much, much lower. A forbidden transition might take milliseconds or even seconds instead of nanoseconds.

These forbidden transitions create metastable states. The electron is in an excited state, but the only way down violates selection rules, so the electron gets stuck there for a while.

Different atoms and molecules have different arrangements of energy levels. In most materials, all the downward transitions are allowed, so electrons fall back down quickly, and you can’t build up population inversion. But in certain materials, ruby, helium-neon, CO₂, rare earth ions, and others, the energy level structure happens to have states where the downward transitions are forbidden or discouraged. These materials can support metastable states.

Finding materials with the right energy level structure is part science, part luck. Researchers calculate energy levels using quantum mechanics, measure them experimentally, and test which materials might work as laser media. Not every material works. In fact, most don’t. The material needs the right combination of properties: energy levels that can be easily pumped, a metastable state with a long enough lifetime, a lasing transition that emits at a useful wavelength, and the material needs to be transparent to both the pump light and the laser light.

How Ruby Works: A Three-Level System

Let’s look more closely at ruby to understand how metastable states work in practice. Ruby is aluminum oxide (Al₂O₃) (essentially sapphire) with about 0.05% of the aluminum atoms replaced by chromium atoms (Cr³⁺). The term “doped” means that small amounts of one element are mixed into the crystal structure of another material. In this case, chromium is doped into aluminum oxide. The chromium atoms are the ones that actually produce the laser light.

Here’s how the energy levels work in chromium:

The chromium ion has a ground state where the electrons are in their lowest energy configuration, happy and stable. It has several high energy levels that electrons can be excited to, these are called the pump bands because they’re where the pump light sends the electrons. And crucially, it has a metastable state sitting in between the ground state and the pump bands.

When you shine bright light from the flash lamp onto the ruby, the chromium ions absorb photons in the blue and green parts of the spectrum. Why blue and green? Because the energy of blue and green photons happens to match the energy difference between the ground state and the pump bands. The electrons jump from the ground state up to those high energy pump bands. So far so good, you’re pumping energy into the system.

Now here’s the clever part: those high energy pump band states are not metastable. Electrons in those pump bands very quickly (in picoseconds) drop down to the metastable state. But this transition doesn’t emit light. Instead, the energy is transferred to vibrations in the crystal lattice, the atoms in the ruby crystal start vibrating more. This is heat. The crystal gets a little warmer, but the electron has moved from the pump band to the metastable state.

And now the electron sits there in the metastable state. It can stay there for milliseconds, which is an eternity on atomic timescales. This is plenty of time for you to pump more and more chromium ions into this metastable state. The flash lamp keeps firing, more electrons jump to pump bands, they drop to the metastable state, and the population in the metastable state builds up.

Eventually, you can get more chromium ions with electrons in the metastable state than in the ground state. You’ve achieved population inversion between the metastable state and the ground state.

Now, when one electron spontaneously drops from the metastable state to the ground state, it emits a red photon at 694.3 nanometers. That photon travels through the crystal, and if it encounters another chromium ion with an electron in the metastable state, it can trigger stimulated emission. Now you have two identical red photons. Those two photons can trigger more stimulated emission. The process cascades, and you get your laser.

This is called a three-level system: ground state, pump bands at high energy, and metastable state at intermediate energy. The key is that you pump to the high levels, electrons quickly relax to the metastable state without emitting light, and then they lase from the metastable state down to the ground state.

Other Laser Media: Different Tricks, Same Principle

Different laser materials use different energy level schemes, but they all rely on having some kind of metastable state or equivalent.

In a helium-neon laser, helium atoms are excited by an electrical discharge, high voltage causes electrons to flow through the gas, colliding with and exciting the helium atoms. The excited helium atoms have energy levels that happen to closely match certain excited levels in neon. When an excited helium atom collides with a neon atom, it can transfer its energy through a collision, leaving the neon atom in an excited state. These excited neon states are metastable. Population inversion builds up in the neon, and the neon lases.

In a CO₂ laser, nitrogen molecules are excited by the electrical discharge. The excited nitrogen molecules are very stable in certain vibrational states and hold onto their energy for a long time. When they collide with CO₂ molecules, they transfer energy to a specific vibrational mode of the CO₂ molecule, the molecule vibrates in a way where the three atoms move asymmetrically. This vibrational mode is metastable. The CO₂ molecules build up population inversion and lase in the infrared.

In semiconductor lasers, the physics is a bit different because you’re dealing with bands of energy levels rather than discrete atomic levels, but the principle is the same: you create conditions where there are more electrons in higher energy states ready to emit photons than there are in lower energy states ready to absorb them.

In fiber lasers, rare earth ions like ytterbium or erbium are embedded in glass, again, this is called doping, where small amounts of these rare earth elements are mixed into the glass material. These ions have metastable states. You pump them with other lasers, electrons move to metastable states, population inversion builds up, and they lase.

The unifying principle is always: you need metastable states or something equivalent so that you can build up population inversion faster than spontaneous emission depletes it. Without metastable states, building a laser becomes extraordinarily difficult or impossible.

Part 6: Where Does the Energy Come From? (Conservation Laws Still Apply)

If a laser beam is incredibly powerful and intense, where is all that energy coming from? Energy can’t be created from nothing, so what’s the source?

The answer is straightforward but important to understand: all the energy in the laser beam comes from whatever is pumping the laser. You’re not creating energy. You’re converting energy from one form to another and concentrating it.

Energy Bookkeeping

Let’s trace the energy flow in a ruby laser:

You have a flash lamp powered by electricity. Let’s say you put in 1000 joules of electrical energy into the flash lamp capacitor. The flash lamp converts that electrical energy into light, though not with perfect efficiency. Maybe 50% becomes light and 50% becomes heat in the flash lamp itself. So now you have 500 joules of light energy radiating from the flash lamp in all directions.

This light shines on the ruby rod. The ruby absorbs some of this light, specifically the blue and green wavelengths that match its absorption bands. But not all the light is absorbed. Some reflects off the surface, some passes through without being absorbed, some is the wrong wavelength for the chromium atoms to absorb. Maybe only 100 joules actually get absorbed by chromium ions in the ruby.

Those 100 joules of absorbed energy excite chromium ions, pumping electrons up to high energy pump band states. Those electrons then drop to the metastable state, giving off the extra energy as heat, the vibrations in the crystal lattice. Then eventually, through stimulated emission, the electrons drop from the metastable state to the ground state, emitting red photons.

But here’s the thing: the red photons emitted have less energy than the blue and green photons that were absorbed. The energy of a photon is proportional to its frequency, and blue light has higher frequency than red light. The energy difference went into heat when the electrons dropped from the pump bands to the metastable state. So maybe only 30 joules actually come out as red laser light.

And of those 30 joules, not all of it comes out as the useful laser beam. Some photons spontaneously emit in random directions and escape from the sides of the rod. Some energy is absorbed by impurities or defects in the ruby. Some energy is lost in the mirrors, which aren’t perfectly reflective. Maybe you get 10 joules out as an actual laser pulse.

So you put in 1000 joules of electricity, and you get 10 joules of laser light. That’s 1% efficiency. The other 990 joules became heat in various parts of the system.

Early ruby lasers were indeed about 1% efficient. Modern fiber lasers have gotten this up to 70-80% efficiency through better designs and materials, which is remarkable, but the principle is the same: you’re not creating energy, you’re converting it from one form (electrical or optical pump energy) to another (laser light), and some is always lost as heat.

What Lasers Actually Do: Energy Concentration

So what is the special thing a laser does? It’s not creating energy from nothing. What it’s doing is concentrating energy spatially, temporally, and spectrally.

Think about it this way: the flash lamp radiates light in all directions. That light is spread out over the entire surface area of the ruby rod and the surrounding space. The light is also spread out over time, the flash lamp might fire for a few milliseconds. And the light is spread out over many wavelengths: blue, green, yellow, all the colors the flash lamp produces, each carrying some of the energy.

The laser takes energy that was spread out in space, time, and wavelength, and concentrates it. The output laser beam is traveling in essentially one direction, not spreading out in all directions like the flash lamp. It’s one wavelength, not spread across the spectrum. And it can be concentrated in time, some lasers produce pulses that are only nanoseconds or even femtoseconds long, which means all that energy is released in an incredibly short duration.

This concentration is what gives lasers their power. A 1-watt laser doesn’t produce more total power than a 1-watt light bulb. Both are converting 1 watt of input energy into light. But the laser concentrates that power into a tiny spot, into a brief duration, into one pure wavelength. That concentration is what allows it to do things like cut through metal.

It’s analogous to how a magnifying glass works. A magnifying glass doesn’t create heat energy from nothing. It takes the sun’s light that would normally be spread over a large area and concentrates it onto a tiny spot. That concentration of the same total energy is what can start a fire, even though no new energy was created.

The laser does something similar, but instead of using a lens to spatially concentrate already-existing light, it uses the physics of stimulated emission to create light that’s already concentrated, monochromatic, directional, and coherent from the moment it’s generated.

Part 7: The Heat Problem (And Why Doesn’t the Laser Melt?)

Another important question: if photons are bouncing back and forth inside the cavity, building up intensity, why doesn’t the laser medium just get incredibly hot or even melt?

The answer has several parts.

Energy Flow in the Laser

First, let’s track where heat is actually generated. You’re pumping energy into the laser medium, whether that’s flash lamp light, electrical discharge, or pump laser light. This energy excites atoms or ions or molecules into higher energy states.

Some of this pump energy goes directly into heat through a process called quantum defect heating. Remember the ruby laser: electrons jump to high pump bands when they absorb blue-green photons, then drop to the metastable state, releasing the difference as heat through vibrations in the crystal lattice. Then they lase from the metastable state, producing red photons. You’re putting in blue-green photons (higher energy) and getting out red photons (lower energy). The difference becomes heat in the material.

Then, not all the excited atoms undergo stimulated emission contributing to the laser beam. Some undergo spontaneous emission, releasing photons in random directions that don’t contribute to the laser beam. These photons are eventually absorbed somewhere in the system and become heat.

The mirrors aren’t perfect. Each time a photon hits a mirror, a tiny fraction is absorbed rather than reflected. This absorption heats up the mirror coatings.

So yes, the laser is constantly generating heat as it operates. Running a laser does heat up the laser medium and the surrounding components. This is a real engineering challenge.

Cooling and Thermal Management

For low-power lasers, like a small helium-neon laser or a laser pointer with a semiconductor laser, the heat generation is small enough that it can be dissipated to the surrounding air through natural convection and radiation. The device might get warm to the touch, but not dangerously hot.

For high-power lasers, cooling becomes critical. Ruby lasers and other pulsed solid-state lasers often use water cooling. The flash lamp and the ruby rod are both cooled by flowing water through channels in the housing. You can only fire the laser in short pulses, and then you have to wait for the heat to dissipate before firing again. Fire too quickly, and the rod will overheat and potentially crack.

CO₂ lasers often have gas flowing continuously through the tube. Fresh, cool gas is constantly supplied at one end, and hot gas is removed at the other end. This convective flow carries away the heat deposited in the gas by the electrical discharge.

For industrial fiber lasers producing kilowatts of continuous power, thermal management is a major part of the design. But fiber lasers have a big advantage here, which we’ll discuss in detail later: their thin geometry means they have a huge surface-area-to-volume ratio, so heat can escape efficiently from the core to the surface.

Why the Intense Light Doesn’t Cause Direct Heating

Now, about the intense light bouncing around inside the cavity: yes, there are billions of photons inside the cavity, but they’re generally not being absorbed by the laser medium in a way that creates heat. That’s the whole point.

Photons at the laser wavelength are essentially transparent to the laser medium when population inversion exists. If a photon encounters an atom with an electron in the ground state, it could potentially be absorbed, but remember, we’ve achieved population inversion, so most atoms have electrons in the excited state, not the ground state. If a photon encounters an atom with an electron in the excited state, it causes stimulated emission, releasing another identical photon. The photon isn’t absorbed and converted to heat; instead, it’s cloned.

So the laser photons bouncing back and forth aren’t directly heating up the medium. They’re either triggering stimulated emission (which adds more photons rather than removing them and converting them to heat) or they’re passing through without interaction.

The heat comes primarily from the pumping process (the quantum defect heating where higher energy pump photons are converted to lower energy laser photons with the difference becoming heat) and from non-ideal efficiencies throughout the system, not from the laser light itself being absorbed by the medium.

That said, in very high-power lasers, there can be issues. If any impurities or defects in the laser medium absorb even a tiny fraction of the laser light, that spot can heat up dramatically and potentially damage the material. This is why high-power laser materials need to be extremely pure and defect-free. Even a tiny impurity that absorbs one part per million of the circulating laser power can create a hot spot that grows and damages the crystal.

The Optical Damage Threshold

There’s also a fundamental limit to how intense light can get before it starts to damage materials through nonlinear effects. At extremely high intensities, the electric field of the light itself becomes so strong that it can rip electrons away from atoms, creating a plasma. This is called optical breakdown, and it permanently damages the material.

So there is an ultimate limit to how much power you can build up inside a laser cavity before you start causing damage. But for most lasers operating under normal conditions, the intensity inside the cavity is well below this damage threshold. The engineering challenge is managing the heat from the pumping inefficiencies, not from the circulating laser light itself.

Part 8: The Evolution of Lasers (Chronological Journey Down the Iceberg)

After Maiman’s ruby laser proved the concept in 1960, researchers realized they could apply the same principles using different materials and different pumping methods. Each new type of laser opened up new applications and solved different problems.

Helium-Neon Laser (1961)

Just a year after the ruby laser, Ali Javan, William Bennett, and Donald Herriott at Bell Labs created the first gas laser using a mixture of helium and neon. This was a major breakthrough because it was the first continuous wave laser, meaning it produced a steady beam rather than pulses.

Instead of a solid crystal, they used a glass tube filled with helium and neon gas at low pressure. Instead of using a flash lamp for pumping, they ran a high voltage through the gas, creating an electrical discharge. If you’ve ever seen a neon sign, you know what this looks like, the gas glows as the electricity flows through it.

The discharge excites helium atoms by giving their electrons enough energy to jump to higher energy levels through collisions with the flowing electrons. These excited helium atoms then collide with neon atoms. Due to a fortunate coincidence, certain excited energy levels in helium have almost exactly the same energy as certain excited levels in neon. When an excited helium atom collides with a ground-state neon atom, the energy can transfer from the helium to the neon through what’s called collisional energy transfer, leaving the neon atom in an excited state.

These excited neon states are metastable, so the neon atoms accumulate in these excited states, creating population inversion. The neon atoms then undergo stimulated emission, producing red light at 632.8 nanometers, which is the characteristic color of helium-neon lasers.

A helium-neon laser. The orange glow in the tube is the gas discharge; the actual laser beam is the thin red line exiting to the left. Photo: Wikimedia Commons, CC BY-SA 3.0.

Helium-neon lasers became the workhorses of many applications because they were relatively cheap, reliable, and produced a stable, continuous beam. For decades, the laser pointer in every physics lecture hall and the barcode scanner in every grocery store used HeNe lasers. They’re still used today for alignment tasks in laboratories and in surveying equipment, though they’re increasingly being replaced by semiconductor lasers which are even cheaper and more compact.

CO₂ Laser (1964)

Kumar Patel at Bell Labs developed the carbon dioxide laser, and it turned out to be incredibly powerful and efficient compared to earlier lasers. This laser uses a gas mixture of carbon dioxide, nitrogen, and helium.

The nitrogen molecules in the mixture are excited by an electrical discharge, similar to the HeNe laser. Nitrogen molecules have vibrational modes (the two nitrogen atoms can vibrate like they’re connected by a spring) that can store energy. When nitrogen molecules are excited into a particular vibrational mode, they hold onto that energy for a very long time because there’s no easy way for the molecule to release the energy. The transition that would allow the nitrogen to de-excite is quantum mechanically discouraged.

When these excited nitrogen molecules collide with CO₂ molecules, they can transfer their energy to a specific vibrational mode of the CO₂ molecule. The CO₂ molecule can vibrate in several different ways, the two oxygen atoms can move symmetrically outward and inward (symmetric stretch), they can move asymmetrically with one moving out while the other moves in (asymmetric stretch), or the molecule can bend. The nitrogen excites the CO₂ into the asymmetric stretch mode, and this mode is metastable enough to build up population inversion.

The CO₂ molecules then lase when they drop from the asymmetric stretch mode to lower vibrational states, producing infrared light at 10,600 nanometers. You can’t see this with your eyes (it’s far into the infrared) but it’s incredibly useful for industrial applications.

CO₂ lasers can be extraordinarily efficient. They can convert 20% or more of the input electrical energy into laser light, which is far better than most other laser types at the time. They can also produce enormous amounts of power. Industrial CO₂ lasers can output tens of kilowatts continuously, and the most powerful ones can exceed 100 kilowatts.

The infrared wavelength is strongly absorbed by most organic materials and many metals, making it ideal for cutting and welding. If you’ve ever seen a laser cutter at a makerspace slicing through plywood or acrylic, there’s a good chance it’s a CO₂ laser. Industrial CO₂ lasers cut through steel plates inches thick. They’re used extensively in manufacturing, in the automotive industry, in aerospace, anywhere precision cutting is needed. A 10-kilowatt CO₂ laser (which would fit in a room about the size of a garage) can cut through 25mm of steel. That’s an inch of solid steel, sliced like butter.

An industrial laser cutting through metal. Photo: Wikimedia Commons, CC BY-SA 3.0.

Dye Lasers (1966)

Peter Sorokin and J. R. Lankard at IBM, working independently from F. P. Schäfer in Germany, created lasers using organic dye molecules dissolved in liquid solvents. This was a completely different approach from solid crystals or gases.

A dye laser uses a solution of organic dye molecules. These are complex molecules with lots of carbon rings, similar in structure to the colorful dyes used in fabric or ink, in fact, some laser dyes are related to textile dyes. The dye gives the solution a vivid color because it absorbs certain wavelengths of light. This dye solution flows through a glass cell, continuously circulated by a pump to prevent overheating in one spot.

You pump the dye optically, either with another laser or with a flash lamp, shining intense light into the cell. The dye molecules absorb the pump light, exciting electrons to higher energy states. What makes organic dye molecules special is their complex molecular structure. They have many closely-spaced energy levels, almost forming a quasi-continuum rather than discrete levels. When excited electrons fall back down, they can emit photons over a range of wavelengths rather than just one specific wavelength.

Here’s what makes dye lasers unique and valuable: by placing wavelength-selective optical elements in the laser cavity (specifically, diffraction gratings or prisms that reflect different wavelengths at different angles) you can select which wavelength gets amplified. By rotating the grating, you can tune the laser’s output wavelength. Want blue light? Rotate the grating one way. Want yellow? Rotate it another way. You can smoothly tune across a range of wavelengths, typically spanning 30-50 nanometers for a given dye.

Different dyes cover different wavelength ranges. Rhodamine 6G, one of the most common laser dyes, can be tuned from about 560 to 640 nanometers, covering yellow-orange-red. Coumarin dyes cover blue-green wavelengths. By switching dyes and adjusting the cavity, you can access wavelengths from the near-ultraviolet all the way to the near-infrared.

This tunability revolutionized spectroscopy, the study of how light interacts with matter. Researchers could tune the laser to exactly the wavelength needed to probe specific molecular transitions, allowing them to study energy levels, chemical bonds, and molecular dynamics with unprecedented precision. Dye lasers were used extensively in chemistry and physics research, in medical diagnostics, and even in isotope separation for nuclear applications. Though they’ve been largely superseded by tunable solid-state lasers and optical parametric oscillators which are more convenient and don’t require flowing liquid dyes, dye lasers were crucial for several decades and are still used in some specialized applications.

Semiconductor Lasers (1962-1970s)

This is where lasers became small, cheap, and ubiquitous. Robert Hall at General Electric created the first semiconductor laser in 1962, just two years after the ruby laser, but it took another decade of development before they became practical for everyday use.

A semiconductor laser, also called a laser diode, uses a semiconductor material, typically gallium arsenide or related compounds like indium phosphide or gallium nitride. You create a p-n junction, which is the same basic structure used in regular diodes and transistors. On one side of the junction, the semiconductor is doped (meaning impurities are intentionally added) to have excess electrons (n-type), and on the other side, it’s doped to have excess holes, which are the absence of electrons that act like positive charge carriers (p-type).

When you apply electrical current in the forward direction across this junction, electrons from the n-type side and holes from the p-type side are pushed toward the junction. At the junction, electrons and holes meet and recombine. When an electron falls into a hole, it drops from a higher energy state in the conduction band to a lower energy state in the valence band, and the energy difference is released as a photon.

At low current, this process just produces incoherent light, which is how an LED works. But at high enough current density, something remarkable happens. The junction region becomes filled with electrons waiting to recombine and holes waiting to be filled. This creates population inversion between the conduction band and valence band. Now when a photon is emitted, it can trigger stimulated emission from other electron-hole pairs in the junction.

The semiconductor chip itself forms the optical cavity. The end faces of the chip are cleaved along natural crystal planes, creating smooth, flat surfaces. The difference in refractive index between the semiconductor (which has a high refractive index, typically around 3.5) and the air outside creates a partial reflection at these surfaces, not as strong as a metal mirror, but typically around 30% reflective, which is enough. Photons bounce back and forth between these end faces, building up intensity through stimulated emission, until laser light emerges.

Early semiconductor lasers had to be cooled to cryogenic temperatures (liquid nitrogen, 77 Kelvin) to work, which made them impractical. The problem was that at room temperature, too many carriers were thermally excited, making it hard to maintain population inversion. But through the 1960s and 1970s, researchers developed better materials and structures. They created double heterostructure designs, where layers of different semiconductor materials with different band gaps are stacked together. These structures confine the electrons, holes, and light to a tiny active region only a few hundred nanometers thick. This dramatically improved efficiency and reduced the threshold current needed to achieve lasing, eventually allowing the lasers to work at room temperature with practical current levels.

By the 1980s, semiconductor lasers were cheap enough and practical enough for consumer applications. They’re incredibly compact, the entire laser chip is typically smaller than a grain of rice, sometimes just a few hundred micrometers on a side. They’re efficient, often converting 30-50% of electrical energy to light, and can run on battery power. They can be modulated on and off extremely quickly, at gigahertz frequencies, which means you can encode billions of bits per second by rapidly switching the laser.

Semiconductor lasers gave us CD players and DVD players, where the laser reads the microscopic pits encoded on the spinning disk. They gave us fiber optic communication, where laser diodes send pulses of light carrying data through fiber cables at incredible speeds. They’re in laser printers, in barcode scanners, in laser pointers, in computer mice, in range-finding sensors for autonomous vehicles. They’re everywhere. The semiconductor laser is probably the most important laser technology in terms of everyday impact on modern life.

Semiconductor lasers at work: each of these fiber optic cables carries data as pulses of laser light. Photo: Brett Sayles / Pexels

Excimer Lasers (1970s)

These ultraviolet lasers use excited dimers (hence the name “excimer,” a contraction of “excited dimer”) which are molecules that only exist in excited states and fall apart when they emit light.

Typical excimers are combinations of a noble gas and a halogen: argon fluoride (ArF), krypton fluoride (KrF), or xenon chloride (XeCl). In the ground state, these atoms don’t bond to each other. A noble gas like argon doesn’t normally form chemical bonds, and it certainly doesn’t bond with a fluorine atom. But when they’re both excited, they can temporarily bond to form an excimer molecule.

You create these excimers by sending a pulsed electrical discharge through a gas mixture at high pressure. The discharge creates excited atoms and ions, which then bond to form excimer molecules. These excited molecules are inherently unstable. They quickly emit a photon and dissociate back into separate atoms.

Because the lower state doesn’t exist as a stable molecule (the atoms simply don’t bond in the ground state) you automatically have population inversion. All the excimers are in the excited state, and there’s essentially no ground state population to absorb photons. This makes excimer lasers very efficient at achieving population inversion.

Excimer lasers produce ultraviolet light at specific wavelengths depending on which excimer you use: ArF at 193 nanometers, KrF at 248 nanometers, XeCl at 308 nanometers. These ultraviolet wavelengths are absorbed very strongly by most materials and have enough energy to break chemical bonds directly.

This property makes excimer lasers ideal for precise material removal, a process called photoablation. The ultraviolet photons are absorbed in a very thin layer at the surface, and they break chemical bonds, essentially vaporizing the material atom by atom. The most famous application is LASIK eye surgery, where an excimer laser removes microscopic amounts of corneal tissue to reshape the eye and correct vision. The laser can remove tissue with incredible precision, a single pulse might remove only 0.25 micrometers of material, and the process is so precise that it doesn’t damage the surrounding tissue.

Excimer lasers are also crucial in semiconductor manufacturing. They’re used in photolithography, the process of creating the intricate patterns on computer chips. The short ultraviolet wavelength allows for extremely fine features to be patterned, the smaller the wavelength, the smaller the features you can create. Modern computer chips with transistors just nanometers in size are manufactured using excimer lasers.

Part 9: The Deep End: Fiber Lasers (Modern Magic)

Photo: Pexels

Now we’re getting into truly advanced technology. Fiber lasers emerged as practical devices in the 1990s and 2000s as researchers figured out how to build high-power lasers using optical fibers as the gain medium. These represent some of the most sophisticated and powerful lasers available today.

What is a Fiber Laser?

The core concept is elegant: instead of having a separate laser crystal or gas tube, and then coupling the laser beam into an optical fiber for transmission to where you need it, what if the fiber itself was the laser?

A fiber laser uses an optical fiber as the gain medium, the optical cavity, and the beam delivery system all in one. The fiber itself does everything.

The Special Fiber

Take an optical fiber, a thin strand of glass about 125 micrometers in diameter, about the thickness of a human hair. But this isn’t ordinary glass. The core of the fiber, the central region where light travels, is doped with rare earth elements, meaning small amounts of rare earth ions are mixed into the glass during manufacturing.

The most common rare earth dopant is ytterbium. Ytterbium ions (Yb³⁺) have the right kind of energy level structure to work as a laser medium. They have a ground state and a metastable excited state, and the energy difference corresponds to near-infrared light around 1030 to 1100 nanometers.

Erbium (Er³⁺) is another common choice, especially for telecommunications, because erbium lases at 1550 nanometers, which happens to be the wavelength where silica optical fibers have the lowest loss, light can travel the farthest through the fiber at this wavelength. Thulium (Tm³⁺) and holmium (Ho³⁺) are used for longer wavelengths around 2000 nanometers, which are useful for materials processing and medical applications.

The rare earth ions are the active laser medium, analogous to how chromium ions were the active medium in ruby. But instead of a rigid rod-shaped crystal a few centimeters long, the laser medium is now a flexible fiber that can be meters or even hundreds of meters long, coiled up to fit in a compact package.

Pumping the Fiber

You pump a fiber laser using semiconductor laser diodes, the technology I described earlier that’s now cheap and efficient thanks to decades of development. These pump lasers are relatively inexpensive compared to other pumping methods.

The pump lasers emit light at a shorter wavelength than the fiber laser output. For ytterbium-doped fiber, the pump lasers typically emit at 915 or 975 nanometers (which corresponds to absorption bands of ytterbium), and the fiber laser outputs at 1030-1100 nanometers.

You couple these pump lasers into the fiber using special optical components. The pump light travels along the fiber, and as it propagates, the rare earth ions absorb the pump light. Their electrons jump to excited states. As you keep pumping energy in, you build up population inversion along the entire length of the fiber.

Because the fiber can be very long (sometimes 10 meters, 50 meters, or even more) you have a lot of length for the pump light to be absorbed and for population inversion to build up. This is one of the advantages of fiber lasers: you can scale up the power by making the fiber longer, giving you more gain medium.

The Optical Cavity: Fiber Bragg Gratings

At each end of the active fiber, you need mirrors to create the optical cavity. But you can’t just glue metal mirrors onto the end of a fiber. Instead, you write Fiber Bragg Gratings directly into the fiber using a clever technique.

A Fiber Bragg Grating (FBG) is a periodic variation in the refractive index of the fiber core. Imagine making the glass slightly denser in some places and less dense in others, in a regular repeating pattern, like stripes running perpendicular to the fiber axis. You create this pattern by exposing the fiber to ultraviolet light through a special mask or using interfering UV beams. The UV light causes a permanent change in the glass structure, slightly altering its refractive index.

This periodic pattern acts as a highly wavelength-selective mirror through a process called Bragg reflection. Light at the Bragg wavelength (determined by the spacing of the periodic pattern) reflects back strongly because reflections from each period of the grating interfere constructively. Light at other wavelengths passes through with minimal reflection because the reflections interfere destructively.

You write one FBG to be highly reflective (99.9% or more) at the laser wavelength and another FBG to be partially reflective (maybe 90% reflective, 10% transmitting). These FBGs, written directly into the fiber as permanent modifications to the glass structure, act as the mirrors for your laser cavity.

The beautiful thing is that this is all integrated into the fiber. There are no separate optical elements to align. The entire laser (gain medium with rare earth ions, optical cavity defined by FBGs, everything) is built into a single piece of fiber that you can coil up.

How Amplification Works

Once you have population inversion in the ytterbium-doped fiber and the optical cavity defined by the FBGs at each end, the lasing process begins. Some spontaneous emission occurs from the excited ytterbium ions, and photons traveling along the fiber between the FBGs start to build up.

These photons travel along the fiber core, which acts as a waveguide keeping them confined. As they propagate, they pass by excited ytterbium ions and trigger stimulated emission. The light is amplified as it travels through the fiber. It reaches the FBG at one end, which reflects it back. The light travels back through the fiber in the opposite direction, gets amplified again by triggering more stimulated emission, reaches the other FBG, which partially reflects and partially transmits.

The transmitted portion that leaks through the partially reflective FBG is your output laser beam. The reflected portion continues bouncing back and forth, continuously building up intensity until a steady state is reached where the gain from stimulated emission equals the losses from the output coupler and various other small loss mechanisms.

Why Fiber Lasers Are Extraordinary

Fiber lasers have several remarkable advantages over traditional bulk lasers, advantages that come directly from their unique geometry and integrated design.

The first major advantage is exceptional heat dissipation. In a traditional rod-shaped laser crystal, heat is generated throughout the volume of the rod when atoms are pumped and when quantum defect heating occurs. This heat has to conduct outward to the surface of the rod to be removed. The rod has a relatively small surface-area-to-volume ratio, which limits how efficiently heat can be extracted. If you try to pump too much power into a bulk crystal, the center heats up more than the edges, creating temperature gradients that cause optical distortions through thermal lensing, and can even crack the crystal from thermal stress.

A fiber, on the other hand, is extremely thin, typically 125 micrometers in diameter. It has an enormous surface-area-to-volume ratio. Heat generated anywhere in the fiber core can quickly conduct the short distance to the surface and dissipate into the surrounding environment. You can coil the fiber and actively cool it very efficiently. This means you can pump much more power into a fiber laser per unit length without overheating problems.

Modern fiber lasers routinely produce multiple kilowatts of continuous power. Industrial fiber lasers are commercially available with output powers of 10 kW, 20 kW, 30 kW, and experimental systems have exceeded 100 kW. To put that in perspective, a 20-kilowatt fiber laser outputs enough power to boil 30 kettles of water every second, except all that power is concentrated into a spot the size of a grain of sand. That’s enough concentrated power to cut through thick steel plates like butter, weld heavy components, or process materials at high speeds. Yet the fiber itself can be kept cool enough that you could touch the outside of the cladding (though the laser beam emerging from it would certainly be extremely dangerous).

The second major advantage is exceptional beam quality. Because the light is guided by the fiber’s waveguide structure, fiber lasers can maintain excellent beam quality. If you use a single-mode fiber, where the core is small enough that only one spatial mode can propagate, the laser output is fundamentally limited to a nearly perfect Gaussian beam with excellent focusability. This beam can be focused through a lens to a tiny spot, concentrating all that power into an incredibly small area.

Traditional high-power bulk lasers often suffer from thermal lensing, where heat-induced refractive index variations distort the beam, and from other aberrations that degrade beam quality as you scale up the power. Fiber lasers don’t have this problem because the light is confined to the small fiber core the entire time, and the excellent cooling keeps the fiber from developing thermal gradients.

The third advantage is remarkable efficiency. Fiber lasers are among the most efficient lasers ever built. They can convert 70% to 80% of the pump power (from the laser diodes) into laser output. Compare this to CO₂ lasers at around 20% efficiency or early ruby lasers at 1% efficiency. This high efficiency means less electricity is wasted as heat, cooling requirements are reduced, and operating costs are significantly lower over the lifetime of the laser.

The fourth advantage is robustness and reliability. Traditional bulk lasers have free-space optics (mirrors, lenses, laser crystals) that need precise alignment. If the laser gets bumped, if temperature changes cause thermal expansion, or if vibrations occur, the alignment can drift and the laser performance degrades or fails entirely. This requires frequent realignment and maintenance.

A fiber laser has no free-space optics to misalign. Everything is built into the fiber with FBGs written directly into the glass. The laser is inherently mechanically stable. It can handle vibration, temperature changes, and harsh industrial environments without performance degradation. Fiber lasers installed in factories can run 24 hours a day, 7 days a week for years with minimal maintenance, often just replacing pump diodes as they age.

And finally, fiber lasers are surprisingly compact for their power output. A multi-kilowatt fiber laser can fit in an enclosure the size of a small refrigerator, whereas a CO₂ laser with similar power might fill an entire room with its gas handling system, high voltage power supplies, cooling systems, and beam delivery optics.

Advanced Fiber Laser Architectures

Once you understand the basic fiber laser, there are several sophisticated variations that push performance even further and enable new applications.

One common architecture is the Master Oscillator Power Amplifier, abbreviated MOPA. Instead of trying to generate all your power in a single laser cavity, you split the system into stages. The master oscillator is a low-power, high-quality fiber laser that generates a clean seed beam with excellent beam quality, narrow linewidth, and well-controlled spectral properties. This seed beam typically outputs milliwatts to watts of power.

This seed beam is then sent through one or more amplifier stages. An amplifier stage is simply a length of rare-earth-doped fiber that’s pumped by laser diodes, but without FBG mirrors forming a cavity. It doesn’t oscillate or lase on its own. Instead, the seed beam passes through, and as it does, it triggers stimulated emission in the pumped fiber, amplifying the signal.

You can chain multiple amplifier stages together. The first stage might amplify the seed from milliwatts to watts, the second stage from watts to hundreds of watts, and subsequent stages can boost it to kilowatts or even tens of kilowatts. Each stage increases the power while maintaining the beam quality and spectral properties of the master oscillator.

This MOPA architecture gives you much more control over the laser characteristics. You can control the pulse shape, duration, and repetition rate by modulating the master oscillator. You can achieve very high peak powers in pulsed operation. You can maintain excellent beam quality throughout the amplification process. And you can scale to high average powers by adding more amplifier stages.

Another important technique is cladding pumping. In the simple fiber laser I described earlier, both the pump light and the laser light travel through the core of the fiber. But there’s a clever variation that makes pumping much more practical: you launch the pump light into the cladding, the layer of glass surrounding the core.

The way this works requires a special double-clad fiber design. The fiber has a small core (maybe 10-20 micrometers diameter) doped with rare earth ions, surrounded by a larger cladding (maybe 125-400 micrometers diameter), and then an outer cladding with even lower refractive index. The inner cladding acts as a waveguide for the pump light, keeping it confined through total internal reflection.

As the pump light propagates through the cladding, bouncing around at various angles, it periodically passes through the doped core. Each time it passes through the core, some pump light is absorbed by the rare earth ions. Over the length of the fiber, all the pump light gets absorbed even though it’s traveling in the larger cladding rather than the core.

The advantage of cladding pumping is enormous. The cladding is much larger than the core, perhaps 100 times larger in cross-sectional area. It’s much easier to couple pump light into a larger area. You can use less expensive, lower brightness pump diodes. You can combine the output from many pump diodes, coupling them all into the cladding from different angles or from different points along the fiber.

Meanwhile, the laser light still travels only in the small core and comes out with excellent beam quality determined by the small core, even though the pumping happens in the much larger cladding. Cladding-pumped fiber lasers are the standard architecture for high-power applications because they allow efficient use of many pump diodes to achieve very high pump powers.

Finally, there are ultrafast fiber lasers. By carefully designing the fiber laser cavity and adding elements that favor short pulses, such as saturable absorbers that preferentially transmit high-intensity light, or through nonlinear effects in the fiber itself, you can create fiber lasers that emit incredibly short pulses of light. These pulses can be femtoseconds in duration. A femtosecond is one quadrillionth of a second, written as 10⁻¹⁵ seconds. To put this in perspective, there are more femtoseconds in one second than there are seconds in 30 million years. Light travels only 300 nanometers in a femtosecond, about the wavelength of ultraviolet light, roughly 1/300th the thickness of a human hair.

These ultrafast pulses have remarkable properties. Because each pulse is so short in time, but still contains a reasonable amount of energy (microjoules to millijoules), the peak power during the pulse is enormous, megawatts to gigawatts, even though the average power might only be watts. When such a short, intense pulse hits a material, the material doesn’t have time to heat up. The energy is deposited and the pulse is gone before heat can conduct away from the absorption region. This allows for incredibly clean, precise material removal without a heat-affected zone.

Ultrafast fiber lasers are used for precision micromachining, cutting or engraving intricate patterns in metals, ceramics, glass, or polymers with minimal thermal damage. They’re used in medical applications for delicate surgery where you want to remove tissue without heating the surrounding area. They’re used in scientific research to study ultrafast phenomena like the dynamics of chemical reactions, electron motion in materials, or the behavior of molecules on femtosecond timescales.

Part 10: Even Deeper (Because the Rabbit Hole Continues)

Just when you think fiber lasers represent the peak of laser technology, there are even more exotic approaches that push the boundaries of what’s possible.

Coherent Beam Combining

What if you want even more power than a single fiber laser can provide? There are fundamental limits to how much power you can extract from a single fiber before nonlinear optical effects distort the beam or before the intensity becomes high enough to damage the fiber itself. The solution is to combine multiple fiber lasers.

In coherent beam combining, you take multiple fiber lasers and carefully control their relative phases so they’re all perfectly synchronized. Then you combine their beams using special optics. If the phases are matched correctly, the beams interfere constructively when they’re combined, and the electric fields add up in amplitude rather than just in power.

Here’s why this is remarkable: if you have ten fiber lasers, each producing one kilowatt of power, and you just combined them incoherently (phases random), you’d get ten kilowatts total. But if you combine them perfectly coherently with all phases aligned, the electric field amplitudes add up, so the electric field is ten times larger. Since intensity is proportional to the square of the electric field, you get one hundred times the intensity, not just ten times. Ten lasers can give you the intensity of a hundred-kilowatt laser in a focused spot.

Of course, achieving perfect coherent combining is extraordinarily difficult. You need to keep the phases of all the lasers matched to within a small fraction of a wavelength (less than a micrometer) despite vibrations, temperature changes, air currents, and other environmental disturbances. This requires sophisticated electronic feedback systems that continuously measure the relative phases using interferometry and adjust them in real time using phase modulators or by adjusting the current to the lasers.

Researchers have demonstrated coherent combining of dozens of fiber lasers, achieving combined output powers of many kilowatts with excellent beam quality. This is an active area of research, with potential applications in directed energy weapons, long-range power beaming, industrial materials processing, and scientific instruments requiring extremely high intensities.

There’s also spectral beam combining, which is a different approach that’s easier to implement but doesn’t give you the intensity enhancement. You take multiple fiber lasers operating at slightly different wavelengths (maybe spaced a few nanometers apart) and use a diffraction grating to combine them spatially. Each wavelength comes in from a slightly different angle, and the diffraction grating bends them all into the same output direction. The powers simply add up, so ten one-kilowatt lasers give you ten kilowatts total. You don’t get the coherent intensity boost, but you also don’t need to control phases, making it much easier to implement for high power levels.

Free Electron Lasers

These represent the absolute extreme of laser technology. They don’t use atoms or molecules as the gain medium at all. Instead, the electrons themselves, moving at relativistic speeds, emit the light.

A free electron laser (FEL) starts with a particle accelerator, like the kinds used in physics research. You accelerate electrons to extremely high energies, approaching the speed of light, giving them enormous kinetic energy. At these speeds, relativistic effects become important, the electrons gain mass and time dilates for them relative to the stationary laboratory.

These high-energy electrons are then sent through a device called an undulator. An undulator is a long series of alternating magnets creating a periodic magnetic field. The magnets are arranged so that the field alternates direction: north-south, south-north, north-south, and so on, typically with a period of a few centimeters.

As the relativistic electrons pass through this alternating magnetic field, the Lorentz force causes them to oscillate back and forth, wiggling in a sinusoidal path. When charged particles accelerate (and changing direction is acceleration) they emit electromagnetic radiation. That’s just classical electromagnetism, known since the 1800s.

So the wiggling electrons emit radiation. But here’s where it gets clever: the spacing of the magnets, the strength of the magnetic field, and the energy of the electrons are all carefully tuned so that the radiation emitted from each wiggle interferes constructively. The radiation builds up coherently along the path of the electrons as they traverse the undulator.

Moreover, the radiation that’s been emitted can interact back with the electrons through what’s called the ponderomotive force. This causes the electrons to bunch up into microbunches separated by one wavelength of the emitted light. These bunches of electrons then radiate coherently, in phase with each other, like a phased array antenna. The radiation becomes even more intense.

This is stimulated emission, but from free electrons rather than bound electrons in atoms. The process is called self-amplified spontaneous emission (SASE) when the bunching happens spontaneously from noise, or a seeded free electron laser when you inject an external laser to trigger and control the bunching process.

Free electron lasers can produce light at almost any wavelength. By adjusting the electron energy and the undulator period, you can tune the output from microwaves to X-rays. This tunability is unique, no other laser technology can cover such a wide range.

The most powerful and scientifically important FELs produce X-rays, creating the brightest X-ray beams in the world by many orders of magnitude. These X-ray free electron lasers (XFELs) are used at national laboratories for cutting-edge science. They can produce X-ray pulses so intense and so brief (femtoseconds) that they can image individual molecules before the X-rays destroy them. They’re used to watch chemical reactions happen in real time, to study materials at the atomic scale, to determine the structure of proteins, and to probe matter under extreme conditions like those in the cores of planets.

Of course, free electron lasers are absolutely enormous. They require particle accelerators hundreds of meters to kilometers long, arrays of powerful magnets, massive power supplies, and infrastructure that costs hundreds of millions to billions of dollars. The Linac Coherent Light Source at SLAC National Accelerator Laboratory uses a 3-kilometer-long accelerator. These are not something you’ll find in a factory or hospital. But for fundamental research at the frontiers of science, they’re invaluable.

X-Ray Lasers from High Harmonic Generation

Another way to produce X-ray laser light without building a kilometer-long accelerator is through high harmonic generation (HHG). You take an ultrafast laser with very intense pulses (peak intensities of 10¹⁴ to 10¹⁵ watts per square centimeter) and focus it into a gas of atoms.

The electric field of the focused laser is so strong that it’s comparable to the electric field that binds electrons to nuclei in atoms. This field distorts the atomic potential, temporarily suppressing the barrier that keeps electrons bound. An electron can tunnel out of the atom, essentially escaping through the barrier while the laser field is pulling it one way.

Then, half a cycle later, the laser field reverses direction and starts pulling the electron back toward the ion it came from. The electron accelerates in the laser field, gaining kinetic energy. When it crashes back into its parent ion, it recombines, and all that kinetic energy plus the ionization energy is released as a single high-energy photon.

Because the electron can be accelerated to high kinetic energies by the laser field before recombining, the emitted photons can have energies far exceeding the photon energy of the driving laser. If the driving laser is infrared, the emitted photons can be in the extreme ultraviolet or soft X-ray region. These photons are harmonics of the original laser frequency, odd multiples like the 11th harmonic, 13th, 15th, going up to the 100th harmonic or higher.

The remarkable thing is that this process naturally generates coherent radiation because the electron dynamics are driven by the coherent laser field. The emitted high-frequency light maintains phase coherence and can be collimated into a beam.

This process allows you to generate coherent extreme ultraviolet and soft X-ray light using a tabletop system. It’s not as powerful or as versatile as a free electron laser, but it’s vastly cheaper and smaller. High harmonic generation sources are used in research labs around the world for time-resolved spectroscopy, studying ultrafast dynamics in materials, attosecond science where processes are studied on timescales of 10⁻¹⁸ seconds, and imaging with nanometer resolution.

The Future

Laser technology continues to advance in multiple directions. Researchers are developing quantum cascade lasers that use engineered quantum wells with artificial energy level structures designed for specific wavelengths, particularly in the mid-infrared where few other lasers operate. There are topological lasers that use concepts from topological physics to create lasers that are immune to defects and can maintain single-mode operation robustly. There are lasers using new materials like perovskites, which might enable new wavelengths and applications, or two-dimensional materials like graphene that might enable ultra-compact integrated photonic circuits.

Power levels continue to increase. Researchers have built lasers with peak powers exceeding a petawatt, that’s 10¹⁵ watts, more than a thousand times the entire electrical generating capacity of the world, concentrated into a femtosecond pulse. These ultra-high-intensity lasers create electric fields strong enough to accelerate particles to relativistic energies, to create matter-antimatter pairs from pure energy through quantum vacuum fluctuations, and to test fundamental physics in extreme conditions.

Efficiency continues to improve. Beam quality continues to improve. New wavelengths become accessible. New applications emerge in fields we haven’t even thought of yet.

From Einstein’s 1917 theoretical prediction to today’s fiber lasers cutting through steel and free electron lasers imaging molecules at the atomic scale, the laser has come an almost unimaginable distance in just over a century.

Conclusion: The Exponential Trick

Right now, at this exact moment, there are hundreds of millions of lasers operating around the world. In your devices, in factories, in hospitals, in data centers, cutting steel, correcting vision, carrying the internet through fiber optic cables. Every single one of them works because photons can clone themselves.

But here’s what really gets me about lasers:

They’re all using the exact same quantum mechanical trick (stimulated emission) but the ways we’ve figured out how to trigger it are wildly different. You can make a laser by flashing bright light at a pink crystal. Or by running electricity through a gas. Or by dissolving colorful dye molecules in liquid. Or by injecting current into a semiconductor chip smaller than a grain of rice. Or by doping rare earth ions into a thin glass fiber. They look nothing alike. They’re built from completely different materials using completely different methods. But at the quantum level, they’re all doing the same thing: convincing atoms to make synchronized copies of photons.

It’s like discovering that the same magic spell works whether you cast it with ruby, neon, organic molecules, or ytterbium ions. The universe doesn’t care about the implementation details. If you can create population inversion and trap the light, stimulated emission will happen. Because that’s just how quantum mechanics works.

And then there’s the exponential part, which is both beautiful and terrifying.

One photon becomes two. Two become four. Four become eight. Exponential growth is always consequential in physics. It’s what makes nuclear chain reactions possible, one neutron splits an atom, releases more neutrons, each splits more atoms. It’s what makes epidemics spread. It’s what makes compound interest powerful. Exponentials don’t mess around.

In a laser, you’re harnessing exponential amplification of light. You start with almost nothing (a few random photons going in the right direction) and within nanoseconds you have trillions of synchronized photons carrying kilowatts of power. That’s the same mathematical process that makes atomic bombs work, except instead of neutrons splitting atoms, you have photons cloning themselves.

We’ve become so good at controlling this exponential process that we can build lasers with peak powers exceeding a petawatt, more power than every power plant on Earth combined, compressed into a femtosecond. We can focus that power to create conditions that only exist in the cores of stars. We’re using it to attempt to ignite fusion reactions. The National Ignition Facility uses 192 lasers to compress hydrogen fuel, and they’ve achieved fusion ignition, the first time humans have created a fusion reaction that produces more energy than it consumes.

Inside the National Ignition Facility target chamber. Each of those circular ports delivers a laser beam. The technician gives you a sense of scale. Photo: Lawrence Livermore National Laboratory.

Exponential amplification of light. It sounds almost innocent until you realize what it means.

Einstein discovered this process in 1917 while trying to make some equations balance. Forty-three years later, someone figured out how to make it happen on purpose. And now we’re using it for everything from reading barcodes to potentially powering civilization.

From abstract quantum mechanics to cutting through steel to maybe, just maybe, unlimited clean energy.

Not bad for some synchronized photons.

Want to go even deeper? Explore: quantum cascade lasers, optical parametric oscillators, Raman lasers, terahertz lasers, quantum dot lasers, polariton lasers, superradiant lasers, and more. The rabbit hole truly never ends.

Here’s the strange part: I don’t actually want to check anything. No message I’m waiting for, no notification that pulled me in. I just… opened it. Automatically. Like a reflex. I catch myself staring at the home screen, thumb hovering, with no idea what I intended to do.

It’s even weirder when you see it from outside. I’ll be talking with someone, mid-sentence, and they pull out their phone and start scrolling. They don’t seem to notice they’ve done it. I don’t think they’re being rude on purpose. It’s more like their hand just… moved.

I got curious about why this happens. I dug into the research, and I have good news and bad news.

The bad news: you’re not going to willpower your way out of this. The systems driving that behavior are ancient, powerful, and mostly invisible to conscious thought.

The good news: once you understand why your brain does this, you can stop blaming yourself and start designing around it. This isn’t a character flaw. It’s evolutionary biology colliding with teams of engineers who’ve figured out exactly how to exploit it.

But I also wonder: if these systems are this powerful, could they be used for good? Could I get addicted to deep work, or learning, or working out, the way I’m addicted to my phone?

Part 1: The Exploitation Playbook

Open TikTok. Swipe. Meh. Swipe. Meh. Swipe. Okay, that was kind of funny. Swipe. Meh. Swipe. Meh. Swipe. Meh. Swipe. Oh wow, that’s actually amazing.

You just experienced the reward pattern that makes slot machines work. And it’s not an accident.

Consider what TikTok is actually doing. Most videos are mediocre, some are good, and occasionally one is perfect for you. You can’t predict which swipe will pay off: variable rewards. The next video is one thumb movement away, no searching, no deciding, no effort: no friction. The feed never ends, so there’s no natural place to quit: no stopping point. And that video that was almost funny, that post that was almost interesting? Each one feels like progress toward the jackpot: near-misses.

This isn’t just TikTok. Instagram Reels, YouTube Shorts, Twitter feeds, Tinder: same formula. Dating apps turn human connection into a swipe-based slot machine. Even browsing Netflix or YouTube’s homepage is a kind of foraging through mediocre thumbnails hoping to strike gold. The pattern is everywhere once you see it.

Why does it work so well? The answer starts with a psychologist, some pigeons, and casinos that figured out the formula before the science existed.

Part 2: The Science of Why It Works

Skinner’s Pigeons

Photo courtesy of the B.F. Skinner Foundation

In the 1950s, B.F. Skinner was studying how animals learn. He put pigeons in boxes with a lever they could peck. Peck the lever, get food.

As Skinner later told it, he stumbled onto something unexpected while running low on food pellets. When rewards came predictably (every peck, or every tenth peck) the pigeons behaved predictably too. They’d peck, eat, take breaks. Sensible birds.

But when rewards came randomly? Sometimes on the third peck, sometimes on the twentieth, sometimes twice in a row? The pigeons became obsessive, pecking with manic persistence, far more than under any predictable schedule. Skinner had discovered variable ratio reinforcement: random rewards for repeated actions.

In 1957, Skinner and his colleague C.B. Ferster published Schedules of Reinforcement, which formalized these findings through rigorous experimentation. But when Skinner wrote about gambling in 1953, he noted something interesting: “the efficacy of such schedules in generating high rates has long been known to the proprietors of gambling establishments.”

The casinos had figured it out empirically before the science caught up.

How Casinos Engineered Addiction

How did casino owners discover the optimal formula for addiction without running controlled experiments on pigeons? To answer that, we need to go back before slot machines existed at all.

In 1891, Sittman and Pitt of Brooklyn, New York built a gambling machine that used five spinning drums, each holding ten playing cards. It was essentially mechanized poker. Players would insert a nickel, pull a lever, and hope the drums landed on a good poker hand. There was no automatic payout; you’d show your result to the bartender and collect your prize (usually free drinks or cigars). The machine was wildly popular. Within a few years, nearly every bar in New York had one.

But notice what this machine didn’t have: drama. All five drums stopped at roughly the same time. You either got a good hand or you didn’t. There was no “almost.”

A few years later, Charles Fey, a Bavarian immigrant working as a mechanic in San Francisco, saw an opportunity. He’d been tinkering with coin-operated machines for years, and in the late 1890s he built something different: the Liberty Bell.

Fey’s design made three changes. First, he replaced the five poker drums with three simpler reels, each showing just five symbols: horseshoes, diamonds, spades, hearts, and liberty bells. Fewer combinations meant automatic payouts became possible.

Second, and more importantly, he made the reels stop sequentially. The first reel lands: cherry. The second reel lands: cherry. The third reel is still spinning… then it lands: lemon. You were so close.

This delay was the innovation that mattered. Fey had accidentally built the near-miss into the machine’s physical structure. Earlier color-wheel machines revealed results all at once, so you could calculate your odds from what you saw. With three sequential reels showing just a fraction of the thousand possible combinations (10 × 10 × 10), players had no way to calculate the payout percentage. More importantly, the sequential stopping created suspense and that feeling of “almost winning” the poker machines lacked.

The Liberty Bell was enormously successful. Because gambling was illegal in California, Fey couldn’t patent his device, and competitors immediately began copying it. Within a decade, slot machines were everywhere.

Photo: Wikimedia Commons

But the real engineering came later. In the early 1980s, when computerized slots began replacing mechanical reels, designers gained precise control over probabilities. They developed a technique called “virtual reel mapping,” where the physical symbols you see spinning don’t correspond to the actual odds. A jackpot symbol might appear on the physical reel as often as any other, but in the computer’s virtual reel (which determines outcomes), it appears far less frequently.

More insidiously, designers began using “clustering,” deliberately placing blank stops next to jackpot symbols on the virtual reel. The result: when you lose, you frequently see the jackpot symbol just above or just below the payline. A near-miss. Your brain registers it as “almost,” even though the computer had already determined you’d lost the instant you pulled the lever.

This wasn’t subtle. In 1988, a case came before the Nevada Gaming Commission challenging one manufacturer’s algorithms for generating an artificially high rate of near-misses. The commission ruled that certain techniques were “unacceptable,” but notably, virtual reel mapping that creates near-misses above and below the payline remains legal to this day.

The casinos didn’t need Skinner’s research to design addictive machines. They had something better: direct feedback on what keeps people pulling the lever, refined over a century. What Fey stumbled onto with sequential reels, modern designers have turned into a science.

The Near-Miss Trick

In 2009, Luke Clark and colleagues at Cambridge put volunteers in brain scanners and had them play slot machines. When someone won, their reward circuits lit up, specifically the ventral striatum, the same region that responds to food and sex. When someone almost won (cherry-cherry-lemon), their reward circuits also lit up. Almost as much as for a real win.

Your brain treats “I almost got it!” as genuine progress. This would be sensible if slot machines were skill-based games where getting close meant you were improving. But they’re not. The outcome is determined the instant you pull the lever. Cherry-cherry-lemon tells you nothing about what the next spin will bring.

Yet your brain thinks: “I was so close! Next time!”

A 2001 study by psychologists Jeffrey Kassinove and Mitchell Schare tested near-miss rates of 15%, 30%, and 45% on participants playing a simulated slot machine. The 30% condition produced the greatest persistence, with participants continuing to play significantly longer than in either the 15% or 45% conditions. Modern slot machines are engineered with this in mind.

Now think about your feed. That video that was almost funny. That profile on Tinder that was almost your type. That thread that was almost insightful. Each near-miss keeps you swiping, because your brain registers it as progress toward the jackpot.

Schultz’s Monkeys

Skinner showed what kept animals hooked. But it took another few decades to understand how the brain actually processes these rewards.

In the 1990s, Wolfram Schultz at Cambridge was studying monkeys. He wanted to understand dopamine, the neurotransmitter everyone assumed was the “pleasure chemical.” The thinking: you eat something delicious, dopamine releases, you feel pleasure. Simple.

Schultz designed an experiment to test this. A monkey reaches into a box, finds a treat. He recorded what the dopamine neurons were doing. At first, the neurons fired when the monkey got the treat. If dopamine equals pleasure, this confirmed the theory.

But then he noticed something that didn’t fit. After the monkey learned that the box always contained a treat, the dopamine response shifted. Now the neurons fired when the monkey saw the box, not when it got the treat. The actual reward barely registered anymore.

And when the monkey expected a treat but didn’t get one? The dopamine neurons went below baseline, a negative signal. When it expected nothing but got a treat anyway? A huge spike.

Schultz realized he wasn’t looking at a pleasure signal. He was looking at a prediction error signal. In a 1997 paper that changed the field, he showed that dopamine tracks the gap between what you expected and what you got:

• Better than expected → dopamine spike
• As expected → nothing
• Worse than expected → dopamine dip

This explains why your tenth bite of cake is less exciting than your first, even though the taste is identical. The first bite exceeded prediction. The tenth matched it. Same cake, different signal.

And it explains why variable rewards are so compelling. When you can’t predict whether the next swipe will be good or bad, every swipe generates potential prediction error. Your dopamine system stays engaged, perpetually anticipating the possibility of surprise.

Berridge’s Rats

Around the same time Schultz was studying monkeys, Kent Berridge at the University of Michigan was studying rats and sweet tastes. Normal rats, when you give them sugar water, make a characteristic facial expression, a “yum” face that looks remarkably similar across mammals, including human babies.

Berridge tried something extreme: he destroyed most of the dopamine system in these rats. If dopamine was the pleasure chemical, these rats shouldn’t enjoy sugar anymore.

But they still liked sugar. When it touched their tongues, they made the same “yum” face. The pleasure was intact. What was gone was any motivation to seek it out. They’d walk right past a pile of sugar and starve to death. If you put sugar in their mouths, they’d happily consume it. They just wouldn’t go get it.

Berridge had discovered that wanting and liking are separate systems. Dopamine doesn’t make you enjoy things. It makes you want things, what Berridge calls “incentive salience,” the feeling that something is worth pursuing. Actual enjoyment comes from different, smaller neural circuits involving opioids.

This separation is what makes certain behaviors so insidious. You can want something intensely while barely enjoying it when you get it.

Think about checking your phone. You feel a pull to check it. That’s wanting, dopamine signaling that something potentially rewarding might be there. You check. Mostly nothing interesting. You don’t particularly enjoy the experience. But a few minutes later, you feel the pull again. The wanting returns, even though the liking never showed up.

Or think about eating chips from a bag. You’re not savoring each chip. They’re fine. But you keep reaching for the next one, and the next one, and suddenly the bag is empty and you feel vaguely sick. The wanting drove the behavior. The liking was barely involved.

Berridge has described addiction as “a starved want in an unstarved brain,” the wanting mechanism running at full intensity while the liking mechanism provides no corresponding satisfaction. App designers don’t need you to enjoy their product. They just need you to keep wanting to check it.

The Foraging Brain

Why does your brain fall for this? Because these systems weren’t designed for the modern world. They were designed for finding food when you don’t know where it is.

The idea that our dopamine systems are essentially foraging circuits comes from optimal foraging theory, developed by ecologists Eric Charnov and Robert MacArthur in the 1960s. Researchers at Xerox PARC later applied this framework to human information-seeking behavior. And ethologist Niko Tinbergen’s work on “supernormal stimuli” helps explain why digital environments are so compelling: artificially exaggerated triggers hijack instincts more effectively than the natural stimuli those instincts evolved for.

Imagine you’re a proto-human on the African savanna. Food is scattered unpredictably across the landscape. Some bushes have berries, most don’t. Some areas have tubers, but you have to dig to find out. Some paths lead to watering holes where you might catch prey, or you might waste hours finding nothing.

Most of your attempts fail. And they have to fail. If food were easy to find, it would already be gone. Your survival depends on persistence through endless disappointment.

The dopamine system evolved to solve this problem. It rewards the search, not just the find. It makes the possibility of finding something almost as motivating as actually finding it. Your ancestors who kept exploring (one more bush, one more trail, one more dig) occasionally hit jackpots: a carcass, a honeycomb, a patch of ripe fruit. Those who gave up too easily starved.

But there’s a second adaptation that’s crucial to understanding why these systems are so exploitable. If every failed attempt felt as bad as a successful find felt good, you’d be too demoralized to continue after three empty bushes. So the brain evolved an asymmetry: wins register strongly, losses barely register at all.

Finding food creates an intense, memorable spike. Not finding food is just… neutral. Not painful. Just nothing. You shrug it off and keep searching.

This asymmetry is exactly what gambling exploits. You lose ten times, and each loss barely registers. Then you win once, and the spike feels significant. Your brain does bad accounting: the one win looms larger than the ten losses. You keep playing.

These systems are ancient. Dopamine-based reward circuits aren’t unique to humans; they exist in insects, fish, even worms. When you feel the compulsive urge to check your phone, you’re fighting hundreds of millions of years of optimization.

But the foraging environment had built-in limits. You had to walk miles. You could only carry so much back to camp. You could only eat so much before you were full. The search eventually ended.

Modern technology has removed all these limits. “Rewards” are infinite. Effort is negligible, just moving your thumb. There’s no upper bound, no natural stopping point. Your brain cannot tell the difference between “I found food that will keep my family alive” and “I found an entertaining video.” Both trigger the same ancient system. Both feel like something. One matters. The other just consumed twenty minutes of your life.

Part 3: What’s New

Skinner’s research is from the 1950s. Schultz’s key papers came out in 1997. You might wonder: have we learned anything new? We have, and it makes the picture more concerning.

TikTok’s algorithm isn’t Skinner’s random reinforcement schedule, and it’s not a slot machine’s fixed probabilities. It’s something more sophisticated: the slot machine is learning you.

Classic variable reinforcement is random. The pigeon gets food on an unpredictable schedule, but the schedule isn’t personalized. A slot machine pays out based on fixed probabilities that apply to everyone equally.

TikTok’s For You page is different. It’s watching which videos make you pause, which ones you watch to the end, which ones you rewatch, which ones you skip past immediately. It’s building a model of your specific dopamine triggers and then optimizing for them. This is variable rewards calibrated to your prediction error system.

This is why people describe the For You page as eerily accurate, like the app knows them better than they know themselves. It’s not just showing you random content hoping something lands. It’s testing, learning, and refining what works for you specifically.

Research on TikTok has identified something called the “flow experience,” a state of absorption where you lose track of time and become fully immersed in the content stream. A 2024 study found that the key predictor of problematic use isn’t just variable rewards; it’s this trance-like concentration where minutes slip into hours without awareness. Users consistently underestimate time spent on TikTok more than on other platforms.

This isn’t academic speculation anymore. In October 2024, thirteen U.S. states and D.C. sued TikTok, alleging that its algorithm is “designed to promote excessive, compulsive, and addictive use” in children. Whether or not the case succeeds, the fact that it exists signals something: the mechanisms we’ve been discussing have moved from psychology journals into courtrooms.

Part 4: Breaking Free

So what do you do with this knowledge?

The first thing to understand: willpower is not the answer. You’re not going to think your way out of systems that operate below conscious awareness and have been optimized for hundreds of millions of years.

But you can change the environment.

Understanding Your Triggers

Cognitive Behavioral Therapy (CBT) helps people identify the thoughts and situations that trigger problematic behaviors. Rather than suppressing urges through willpower, CBT focuses on noticing patterns: what happens right before the behavior? What internal state are you in? What are you trying to avoid or achieve?

In trials with smartphone-addicted adolescents, 12-week CBT programs significantly reduced addiction scores. The most helpful module, according to participants, was called “Recognize the Triggers,” which taught people to notice why they were reaching for their phone in the first place.

The insight is that phone checking isn’t random. It’s triggered by specific internal states: boredom, loneliness, anxiety, the desire to escape an uncomfortable task, the need for stimulation. If you can notice why you’re reaching for the phone, you create a moment of choice that wasn’t there before.

Some questions to ask yourself in that moment: Am I feeling bored? What am I avoiding? Am I feeling lonely? What connection am I actually seeking? Am I feeling anxious? What would actually address the anxiety? And perhaps most useful, given what we know about wanting versus liking: will I actually enjoy this, or am I just responding to a pull?

That last question is worth developing into a practice. Before you check your phone, predict: “Will I feel better after 10 minutes of scrolling?” Then afterward, notice whether your prediction was accurate. Building this awareness helps reveal the gap between the pull and the payoff, the wanting that persists even when liking never shows up.

Environmental Design

The more effective approach is changing the environment so the behavior becomes harder in the first place. You won’t out-think systems optimized for millions of years. But you can redesign choice architecture so checking your phone requires more effort than not checking it.

The simplest intervention: turn off notifications. Every notification is a trigger, every buzz is your phone asking for attention. Most apps default to aggressive notification settings because their metric is engagement, not your wellbeing. Turn off notifications for everything except calls and messages from actual humans, and you’ve eliminated hundreds of daily triggers.

The next step is removing apps from your home screen. If you have to search for an app to open it, you’ve added a few seconds of friction. That small delay creates a moment where you can ask “Do I actually want to do this?” Having Instagram’s icon staring at you every time you unlock your phone is a cue that triggers wanting. Remove the cue and you remove the trigger.

For deeper focus, put your phone in another room entirely. A 2017 study from UT Austin found that participants with their phones in another room significantly outperformed those with phones on the desk, even when the phones were face-down and silent. Just having the phone nearby seemed to occupy cognitive resources. Part of your brain was dedicated to not checking it, and that effort cost something. (A 2024 meta-analysis of 33 studies found smaller effects than the original study, suggesting this may vary by individual, but even a small effect compounds over time.)

A more aggressive intervention is grayscale mode. A 2024 study by Dekker and Baumgartner found that participants used their phones about 20 minutes less per day with grayscale displays. The mechanism is simple: colorful visuals trigger dopamine responses. App icons and notification badges use bright, saturated colors specifically because those colors grab attention. Remove the colors, and the phone becomes less visually compelling. Some users find grayscale hard to maintain long-term, but even using it intermittently (say, after 9pm) can help.

Other friction techniques include logging out of apps after each use (having to re-enter a password creates a pause that’s often enough to break the automatic behavior) and using browser versions instead of apps (the mobile web version of Twitter or Instagram is clunkier, slower, and less optimized for addiction, which is a feature, not a bug).

For some people, friction isn’t enough. Deleting the apps entirely is the only thing that works. If you find yourself reinstalling apps you’ve deleted, that’s useful information about how strong the pull is.

Making Losses Visible

Remember the foraging asymmetry: wins register strongly, losses fade. One way to counter this is to make the losses visible.

Screen time tracking does this automatically. Seeing “4 hours 23 minutes on TikTok” at the end of the day is information your brain would otherwise ignore. You might not remember the scrolling (each mediocre video faded from memory as soon as it passed) but you’ll notice the number.

Some people find journaling useful: after a scrolling session, write down what you actually got from it. Often the answer is “nothing” or “I feel worse.” Recording this counters the brain’s tendency to remember the occasional good video and forget the hundred forgettable ones.

Part 5: Using It For Good

If these mechanisms are so powerful, can they be used for good? Can you get addicted to working out, or to learning, or to doing deep work? Imagine if school were as compelling as TikTok. Imagine if you felt the same pull toward your workout that you feel toward your phone.

People have tried. The results are instructive.

What Duolingo Got Wrong

Duolingo is the most famous attempt at making learning addictive. If you haven’t used it: the app teaches languages through short exercises, mostly translation. Complete a lesson and earn XP. The amount varies unpredictably: bonus XP for a “perfect lesson” or daily challenges. Leaderboards let you compete against strangers for weekly rankings, and streaks track consecutive days practiced. Miss a day and the streak resets.

The notifications are remarkably persistent. Duo, the green owl mascot, sends messages like “These reminders don’t seem to be working. We’ll stop sending them for now.” (guilt trip) or “You made Duo sad” with an image of the owl looking dejected. According to Duolingo’s own reports, passive-aggressive notifications outperform friendly ones.

And it works, for engagement. Duolingo’s 2024 investor reports show daily active users at 34% of monthly active users, with over 10 million maintaining year-long streaks.

But many people use Duolingo for years and still can’t hold a conversation. Applied linguist Matt Kessler has studied this gap: Duolingo is effective for receptive skills (reading, listening, vocabulary) but users consistently struggle with productive skills like speaking and writing. One user described arriving in Sweden after hundreds of hours on Duolingo, able to read magazine articles but unable to order a coffee.

The problem isn’t just gamification. It’s that Duolingo’s core learning method (translation between languages) trains you to convert rather than to think in the new language. And because streaks reward showing up rather than improving, users optimize for the wrong thing. Someone with a 500-day streak might spend their daily five minutes on an easy lesson they’ve already mastered, just to preserve the streak, when watching a three-minute video in the target language would be more useful.

Duolingo made people addicted to using the app, not to learning. The engagement and the outcome became decoupled.

What About “TikTok for Learning”?

You’ve probably seen ads for apps like Headway, Imprint, or Blinkist: “TikTok for books” or “TikTok for smart people.” Headway offers swipeable book summaries with streaks and gamified challenges. Imprint presents ideas through tap-through visual slides. Users report replacing doomscrolling with these apps.

But notice what they’re doing: they’ve applied TikTok-style engagement to consumption, not skill-building. You’re not learning to speak Spanish or play piano. You’re consuming summaries of productivity books. The engagement mechanics work, but the output is passive absorption, not active skill. It’s watching workout videos instead of working out.

Nobody has applied this level of design sophistication to the hard problem: making skill acquisition feel as compelling as scrolling.

The Gamification Trap

Could you just add variable rewards to beneficial tasks? Imagine an app where you complete small work tasks, and sometimes you get a reward and sometimes you don’t, unpredictably. Would that drive motivation?

A 1999 meta-analysis by Edward Deci and colleagues examined 128 studies on this question. The finding was counterintuitive: when people expect rewards for an activity, their intrinsic motivation for that activity decreases. The effect size was substantial (d = -0.40 for performance-contingent rewards). This is the “overjustification effect,” replicated many times.

The mechanism is attribution. When you’re rewarded for doing something, you start to attribute your behavior to the reward rather than to genuine interest. “I’m doing this because I get points,” not “I’m doing this because I find it interesting.” When the rewards stop, motivation drops below where it started. The reward didn’t just fail to help; it actively undermined the original motivation.

This explains why naive gamification often produces a burst of engagement followed by a fade. Points and badges create short-term excitement, but they’re training you to care about the points, not about the activity. Once the novelty wears off, you’re left with less intrinsic motivation than you had before.

The exceptions are revealing. Unexpected rewards don’t undermine intrinsic motivation, because you can’t attribute your prior behavior to a reward you didn’t know was coming. And informational feedback (“you did really well on that”) can actually enhance motivation, because it provides genuine evidence of competence rather than external control.

This suggests variable rewards could work, but only if they’re genuinely unexpected and the tasks themselves build real skills. The moment people start expecting the rewards, the mechanism shifts from enhancement to undermining.

The Flow Alternative

Maybe the goal shouldn’t be to make learning feel like TikTok. Maybe TikTok is the wrong model entirely.

Addiction implies compulsion despite harm or lack of benefit. You keep swiping even though you’re not enjoying it, even though you have other things to do, even though you’ll feel worse afterward. Wanting without liking.

But people who genuinely love learning or working out describe something different. They’re not compelled despite lack of benefit; they find the activity itself rewarding. Wanting and liking are aligned. They might be deeply engaged, but they’re not trapped.

Psychologist Mihaly Csikszentmihalyi spent decades studying this state, which he called “flow.” It occurs when three conditions are met: clear goals with immediate feedback, challenge matched to skill (hard enough to stretch you, not so hard you’re overwhelmed), and deep concentration. In flow, people lose track of time and feel intrinsically motivated to continue. Absorbing in a way that feels good, not compulsive.

Diagram: Wikimedia Commons

The design principles for flow are different from the design principles for addiction.

First, challenge has to match skill. Too easy and you’re bored; too hard and you’re anxious. The sweet spot is where you’re stretched but capable. This is why personalized difficulty adjustment matters, and why one-size-fits-all content often fails.

Second, feedback needs to be immediate and informational: not “you earned 10 points” but “you got that right” or “here’s what you missed.” The feedback should tell you about your actual performance, not your standing in a game.

Third, real progress has to be visible. Anki does this well: you can see yourself remembering things you used to forget. Couch to 5K does this too: you can run distances that used to be impossible. The reward is the actual improvement, made salient.

Fourth, effective systems build toward production, not just consumption. Duolingo’s weakness is that it trains reception (reading, listening) but not production (speaking, writing). Real skill requires doing the thing, not just recognizing it.

Finally, based on the overjustification research, expected tangible rewards will undermine intrinsic motivation over time. If rewards are used at all, they should be unexpected, or replaced entirely by informational feedback about genuine progress.

What About Work That’s Just… Work?

Flow assumes the activity can become intrinsically rewarding. But some work is genuinely tedious: IT support, data entry, answering the same questions repeatedly. You’re not going to achieve flow while resetting someone’s password for the hundredth time. Can anything from engagement science help?

Probably yes, but not through naive gamification. Adding points and leaderboards to IT support tickets would likely produce the Duolingo pattern: short-term boost, then fade, leaving workers less motivated than before.

What might actually help borrows from TikTok without the extractive parts.

One approach is introducing variability in the task itself, not just rewards. TikTok keeps you engaged partly through unpredictability. For repetitive work, mixing task types or varying the order could break monotony. An IT support queue that surfaces an interesting edge case between routine password resets gives you something to look forward to.

Another is reframing impact. “You closed 47 tickets” is a metric. “You helped 47 people get back to their work today” is a story. Research on meaningful work suggests that connecting tasks to their human consequences increases motivation, even for routine work.

Recognition helps too, but it has to be unexpected, not scheduled. A manager who occasionally notices good work, without a predictable pattern, provides unexpected positive feedback that enhances rather than undermines motivation. The key is that it’s genuinely informational (“that was a really clear explanation”) rather than controlling (“you earned 10 recognition points”).

Autonomy matters even when the task is fixed. Control over how you approach it increases engagement. Letting IT support staff develop their own scripts, templates, or workflows gives ownership over process even when they can’t control the work itself.

And finally, there’s progress toward mastery, even in routine work. Is your average resolution time dropping? Are users rating your explanations more highly? Making this visible transforms “doing the same thing over and over” into “getting better at something.”

None of this makes tedious work feel like TikTok. But it might make it less soul-crushing, and unlike naive gamification, these approaches are less likely to backfire.

The Open Question

Could these principles produce something as compelling as TikTok for genuinely beneficial activities? We don’t fully know, because nobody has tried with TikTok-level resources. The apps that work best (Anki, Couch to 5K) are relatively simple. The apps with TikTok-level engagement (Headway, Imprint) are optimized for consumption, not skill-building.

There may be a fundamental asymmetry. Extractive apps can make the reward arbitrarily easy: just keep swiping, and sometimes something good appears. Skill-building requires actual effort, and there may be no way around that. Flow is deeply rewarding, but you have to earn it.

Still, the gap between what exists and what’s possible feels large. What would a TikTok-level engineering effort look like if aimed at genuine skill acquisition? Personalized challenge-skill matching, immediate informational feedback, visible real progress, variability without expected rewards?

We might not get “addicted to working” in the slot-machine sense. But we might get something better: work that feels meaningful and produces visible results, in ways that make you want to keep doing it. Not compulsion despite lack of benefit, but genuine engagement with genuine payoff.

The Lever

The systems are ancient. The exploitation is modern. And now you know the difference.

You can’t rewire your dopamine system. Hundreds of millions of years of evolution aren’t going to yield to good intentions. But you can design your environment so the easy behaviors are the good ones and the extractive ones require effort. You can notice the gap between wanting and liking, between the pull and the payoff.

Can the same mechanisms be redirected toward things that actually benefit you? Not simply. Naive gamification (points, badges, streaks) tends to undermine the intrinsic motivation it’s trying to enhance. Duolingo can make you addicted to using Duolingo without making you fluent in Spanish.

But there’s a different state worth aiming for. Flow isn’t addiction. In flow, wanting and liking align: you’re deeply engaged and genuinely enjoying the activity and making real progress. The conditions for flow (challenge matched to skill, immediate feedback, clear goals) are different from slot-machine engagement. Harder to engineer, but they don’t leave you empty afterward.

The question isn’t whether to engage your brain’s reward systems. They’re engaged whether you choose it or not.

The question is who’s holding the lever, and whether what it’s pointing at is worth wanting.

Sources and Further Reading

Foundational Neuroscience

• Schultz, W., Dayan, P., & Montague, P.R. (1997). “A neural substrate of prediction and reward.” Science.
• Schultz, W. (1998). “Predictive reward signal of dopamine neurons.” Journal of Neurophysiology.
• Berridge, K.C. & Robinson, T.E. (2016). “Liking, wanting, and the incentive-sensitization theory of addiction.” American Psychologist.

On Variable Reinforcement and Gambling

• Ferster, C.B. & Skinner, B.F. (1957). Schedules of Reinforcement. Appleton-Century-Crofts.
• Clark, L. et al. (2009). “Gambling near-misses enhance motivation to gamble and recruit win-related brain circuitry.” Neuron.
• Harrigan, K.A. (2008). “Slot Machine Structural Characteristics: Creating Near Misses Using High Award Symbol Ratios.” International Journal of Mental Health and Addiction.

On Smartphone and Social Media Effects

• Ward, A.F. et al. (2017). “Brain Drain: The Mere Presence of One’s Own Smartphone Reduces Available Cognitive Capacity.” Journal of the Association for Consumer Research.
• Dekker, C.A. & Baumgartner, S.E. (2024). “Is life brighter when your phone is not? The efficacy of a grayscale smartphone intervention.” Mobile Media & Communication.

On TikTok and Modern Platforms

• “Does TikTok Addiction exist? A qualitative study.” PMC (2024).
• Brown University. “What Makes TikTok so Addictive?”

On Treatment and Behavior Change

• “Interventions for Digital Addiction: Umbrella Review of Meta-Analyses.” Journal of Medical Internet Research (2025).

Ethical Design

• Center for Humane Technology. “humanetech.com”

Many developers using AI coding assistants encounter this. While the AI understands the code’s intent, translating that understanding into precise, automated file modifications presents significant technical challenges.

Why Precise File Editing Matters

Effective file editing is central to the value proposition of coding assistants. If these tools cannot reliably modify code files, their utility diminishes, reducing them to suggestion engines that require manual implementation. An assistant capable of dependable automated editing saves developers considerable time and cognitive load compared to one prone to failures.

The fundamental challenge lies in the indirect nature of the operation: Large Language Models (LLMs) lack direct file system access. They must describe intended changes via specialized tools or APIs, which then interpret these instructions and attempt execution. This handoff between the LLM’s representation and the file system state is a frequent source of complications.

Users of tools like GitHub Copilot, Aider, or RooCode may have observed these struggles: edits failing to locate the correct insertion point, incorrect indentation, or the tool ultimately requesting manual application.

What You Will Learn

This post examines the file editing mechanisms of several coding assistant systems: Codex, Aider, OpenHands, RooCode, and Cursor. For the open-source systems (Codex, Aider, OpenHands, RooCode), the insights presented here are derived from analyzing their respective codebases. For Cursor, which is closed-source, the insights come from public discussions and interviews with their team. We will explore their approaches, presented roughly in order of increasing complexity, while noting that their development involved parallel evolution and mutual influence.

For each system, we will analyze:

1. How it receives edit instructions from the AI.
2. How it interprets and processes these instructions.
3. How it applies the changes to files.
4. How it handles errors and edge cases.
5. How it provides feedback on the outcome.

Understanding these mechanisms provides insight into the difficulties of automated code editing and the increasingly sophisticated solutions different systems employ.

Key Concepts in AI Code Editing

Before proceeding, let’s define some terms frequently used in this domain:

• Patch: A formal specification of changes (additions, deletions) for a file, often including metadata like file paths and context for application.
• Diff: A format highlighting line-by-line differences between text versions, typically using + and - indicators, focusing on content changes.
• Search/Replace Block: An editing instruction format using delimiters (e.g., <<<<<<< SEARCH, >>>>>>> REPLACE) to explicitly define text to find and its replacement.
• Context Lines: Unmodified lines included in patches or diffs surrounding changes, used to accurately locate the modification point.
• Hunk: A contiguous block of changes within a patch or diff, comprising context lines and modifications.
• Fuzzy Matching: Algorithms (e.g., using Levenshtein distance) to find approximate matches for text strings, handling minor variations.
• Indentation Preservation: Maintaining consistent whitespace prefixes (spaces, tabs) during file edits, critical for syntax and readability.
• Fence: Delimiters (e.g., triple backticks ```) clearly marking the boundaries of code blocks in text or instructions.

The File Editing Workflow

Most AI code editing systems follow a general workflow:

LLM (generates change description) → Tool (interprets & applies) → File System (state change) → Feedback (Tool reports outcome) → LLM (processes feedback)

While straightforward conceptually, several challenges arise in practice:

Challenge 1: Locating the Edit Target

The LLM often operates on a potentially outdated or incomplete view of the target file. Accurately finding the intended edit location becomes difficult when:

• The file has been modified since the LLM last accessed it.
• The file contains multiple similar code sections.
• The file exceeds the LLM’s context window capacity.

Context mismatches are common when the file state diverges. Robust systems provide detailed error feedback, enabling the LLM to adapt. Some may provide the current file state upon error.

Challenge 2: Handling Multi-File Changes

Code modifications frequently span multiple files, introducing complexities:

• Ensuring consistency across related edits.
• Managing dependencies between files.
• Applying changes in the correct sequence.

Most systems address this by processing edits sequentially, file by file.

Challenge 3: Maintaining Code Style

Developers require adherence to specific formatting conventions. Automated edits must preserve:

• Indentation style (tabs vs. spaces, width).
• Line ending conventions.
• Comment formatting.
• Consistent spacing patterns.

Challenge 4: Managing Failures

A robust editing system should handle failures gracefully:

• Provide clear explanations for the failure.
• Offer diagnostic information to aid correction.
• Potentially attempt alternative strategies upon initial failure.

Common Edit Description Formats

AI systems use various formats to communicate intended changes:

1. Patches: Detailed add/delete instructions, often based on standard patch formats.
2. Diffs: Showing differences between original and desired states.
3. Search/Replace Blocks: Explicitly defining find/replace operations.
4. Line Operations: Specifying edits by line number (less common due to fragility).
5. AI-Assisted Application: Employing a secondary AI model specifically for applying complex changes.

Let’s examine how specific systems implement these concepts.

Codex: A Straightforward Patch-Based System

OpenAI’s Codex CLI utilizes a relatively simple, structured patch format. Its effectiveness stems partly from OpenAI’s ability to train its models specifically to generate this format reliably.

The Codex Patch Format

The LLM communicates changes using this structure:

*** Begin Patch
*** [Operation] File: [filepath]
@@ [text matching a line near the change]
  [context line (unchanged, starts with space)]
- [line to remove (starts with -)]
+ [line to add (starts with +)]
  [another context line]
*** End Patch

Key features:

• Operation: Add File, Update File, or Delete File.
• @@: Followed by text content from a line near the edit (e.g., function/class definition) used for locating the change. Crucially, this avoids direct reliance on line numbers.
• Context lines (start with space ` `): Must match existing file content and remain unchanged; used for precise anchoring.
• - prefixed lines: Marked for deletion.
• + prefixed lines: Marked for addition.

Consider this example modifying a print statement:

*** Begin Patch
*** Update File: main.py
@@ def main():
   # This is the main function
-  print("hello")
+  print("hello world!")
   return None
*** End Patch

Here, @@ def main(): helps locate the function, while the space-prefixed context lines (# This is..., return None) pinpoint the exact edit location.

The system attempts to match the @@ line and context lines exactly. If this fails, it employs fallback strategies: first matching with trimmed line endings, then matching with all whitespace trimmed. This flexibility accommodates minor discrepancies between the LLM’s view and the actual file. A single patch can contain multiple @@ sections to target different parts of a file.

Patch Parsing and Application

Upon receiving a patch via the apply_patch tool, the system performs these steps:

1. Validates the basic patch structure (*** Begin Patch / *** End Patch).
2. Identifies the target file(s).
3. Loads the current content of the target file(s).
4. Parses the patch into discrete operations (create, update sections, delete).
5. Attempts to apply the changes to the loaded file content.

Fuzzy Matching for Robustness

The progressive matching strategy for context lines enhances robustness:

1. Attempt exact match.
2. If failed, attempt match ignoring line endings.
3. If failed, attempt match ignoring all whitespace.

This helps overcome small variations between the expected and actual file content.

Error Handling and Feedback Mechanisms

Codex provides structured JSON feedback upon failure, aiding the LLM’s correction attempts:

Context Line Mismatch:

{
  "exit_code": 1,
  "stderr": "Error: context line 3, \"  const response = await fetch(`/api/users/${userId}`);\" does not match \"  const response = await fetch(`/api/users/${userId}`, { headers });\""
}

(Clearly indicates the mismatch and the differing lines.)

File Not Found:

{
  "exit_code": 1,
  "stderr": "Error: File not found: src/api.js"
}

(Unambiguous file access error.)

Invalid Patch Format:

{
  "exit_code": 1,
  "stderr": "Error: Patch must start with *** Begin Patch"
}

(Specific format violation feedback.)

This detailed, structured feedback allows the LLM to diagnose the failure and potentially retry with a corrected approach (e.g., re-fetching the file or adjusting the patch context).

Example: Real-World Edit Scenario

Let’s examine how Codex handles a real-world edit scenario. Imagine we need to add error handling to a function that fetches user data:

Original code in src/api.js:

async function fetchUserData(userId) {
  const response = await fetch(`/api/users/${userId}`);
  const data = await response.json();
  return data;
}

LLM generates a patch:

*** Begin Patch
*** Update File: src/api.js
@@ async function fetchUserData(userId) {
-  const response = await fetch(`/api/users/${userId}`);
-  const data = await response.json();
-  return data;
+  try {
+    const response = await fetch(`/api/users/${userId}`);
+    if (!response.ok) {
+      throw new Error(`Failed to fetch user data: ${response.status}`);
+    }
+    const data = await response.json();
+    return data;
+  } catch (error) {
+    console.error(`Error fetching user ${userId}:`, error);
+    throw error;
+  }
 }

@@ function formatUserData(data) {
-  return data;
+  return {
+    id: data.id,
+    name: data.name,
+    email: data.email,
+    formattedDate: new Date(data.createdAt).toLocaleDateString()
+  };
 }
*** End Patch

This example shows a patch that modifies two different functions in the same file, each with its own @@ context marker.

OpenAI’s Patch Format Standardization

With the release of GPT-4.1 (April 2025), OpenAI published a “prompt cookbook” detailing this recommended patch format and a reference implementation (apply_patch.py). They indicated significant training effort for GPT-4.1 on this format, contributing to its effective use within the Codex CLI ecosystem.

OpenAI’s commentary highlighted that successful formats often avoid line numbers and clearly provide both the code to be replaced and its replacement, using distinct delimiters. This suggests core principles for reliable AI-driven editing. OpenAI’s ability to co-develop the LLM and the editing tool allows for tight integration and optimization.

Aider: A Multi-Format Editing System

Aider employs a more flexible approach, supporting multiple edit formats. It can select the format best suited to the task or the specific LLM being used.

Pluggable Edit Format Architecture

Aider uses a system of “coder” classes, each responsible for handling a specific edit format:

class Coder:
    # ... attributes like edit_format identifier ...

    def get_edits(self): # Parses AI response into edit operations
        raise NotImplementedError

    def apply_edits(self, edits): # Applies parsed edits to files
        raise NotImplementedError

This modular design allows for easy extension and selection of different editing strategies.

Supported Edit Formats in Aider

Aider supports several formats, choosing based on the model or user configuration (--edit-format):

1. EditBlock Format (Search/Replace): Intuitive format clearly showing search/replace blocks.

   file.py
   <<<<<<< SEARCH
   # Code block to find
   =======
   # Code block to replace with
   >>>>>>> REPLACE
   

2. Unified Diff Format (udiff): Standard diff format (diff -U0 style), suitable for complex changes.

   --- file.py
   +++ file.py
   @@ -10,7 +10,7 @@
    def some_function():
   -    return "old value"
   +    return "new value"
   

3. OpenAI Patch Format: Aider implemented OpenAI’s reference format, leveraging GPT-4.1’s training on this syntax.

   *** Begin Patch
   *** Update File: file.py
   @@ class MyClass:
       def some_function():
   -        return "old"
   +        return "new"
   *** End Patch
   

4. Additional Formats:

   • whole: LLM returns the complete modified file content. Simple but potentially inefficient for large files.
   • diff-fenced: Diff format variant where the filename is inside the code fence (```), used with models like Gemini.
   • editor-diff / editor-whole: Streamlined versions for specific internal modes.

Flexible Search Strategies

When applying Search/Replace blocks, Aider attempts multiple matching strategies sequentially:

1. Exact match.
2. Whitespace-insensitive match.
3. Indentation-preserving match.
4. Fuzzy match using difflib.

This layered approach increases the likelihood of successfully applying edits even with minor imperfections in the SEARCH block.

Detailed Error Reporting

Aider excels at providing highly informative feedback when edits fail:

# 1 SEARCH/REPLACE block failed to match!

## SearchReplaceNoExactMatch: This SEARCH block failed to exactly match lines in src/api.js
<<<<<<< SEARCH
async function fetchUserData(userId) {
  const response = await fetch(`/api/users/${userId}`);
  const data = await response.json();
  return data;
}
=======
...
>>>>>>> REPLACE

Did you mean to match some of these actual lines from src/api.js?


async function fetchUserData(userId) {
    const response = await fetch(`/api/users/${userId}`);
    // Some comment here
    const data = await response.json();
    return data;
}

The SEARCH section must exactly match an existing block of lines including all white space, comments, indentation, docstrings, etc

# The other X SEARCH/REPLACE blocks were applied successfully.
Don't re-send them.
Just reply with fixed versions of the blocks above that failed to match.

This feedback is significantly more detailed than simple failure messages. It explains the mismatch, suggests potential correct targets, reiterates the matching rules, and instructs the AI on how to proceed (only resend failed blocks). This detailed guidance greatly improves the AI’s ability to correct its edits.

While adopting OpenAI’s format, Aider enhances it with greater flexibility and substantially more informative error handling.

OpenHands: Blending Traditional and AI-Assisted Editing

OpenHands primarily relies on traditional edit application methods while also incorporating an optional LLM-based editing capability.

Traditional Edit Application

OpenHands primarily uses traditional editing approaches. It has built-in support for detecting different patch formats – including unified diffs, git diffs, context diffs, ed scripts, and RCS ed scripts – using regular expression patterns. Based on the detected format, it applies the appropriate parsing and application logic. The system supports several traditional editing methods:

1. String replacement.
2. Line-based operations (by number).
3. Standard patch application utilities.

It includes features like whitespace normalization to handle variations in patch indentation.

Optional LLM-Based Editing Feature

OpenHands allows configuring a separate “draft editor” LLM for a distinct editing workflow:

1. Target Identification: The primary LLM specifies the target line range for the edit.
2. Content Extraction: The tool extracts this specific code section.
3. LLM Rewrite: The extracted section and a description of the desired change are sent to the specialized “draft editor” LLM. This editor LLM can have different configurations (model, temperature) optimized for editing.
4. File Reconstruction: The tool receives the modified section from the editor LLM and integrates it back into the file, replacing the original lines.

To ensure the draft editor LLM produces the correct output for integration, it is given a specific system prompt instructing it to:

• Produce a complete and accurate version of the modified code section.
• Replace any placeholder comments (like # no changes needed before this line) with the actual, unchanged code from the original section if parts were meant to be preserved.
• Ensure correct and consistent indentation is maintained throughout the block.
• Output the final, complete edited content precisely wrapped in a designated code block format for easy parsing by the tool.

Potential benefits of a separate editor LLM:

• Task-Specific Tuning: Optimize parameters specifically for code modification.
• Model Flexibility: Use different models for reasoning vs. editing.
• Focused Prompting: Provide the editor LLM with a narrow, edit-specific prompt.

The reconstruction process carefully combines the content before the edit, the LLM-edited block, and the content after the edit. Optional validation steps like linting can be performed.

This LLM-based editing appears to be an optional, potentially experimental feature within OpenHands, often disabled by default.

RooCode: Advanced Search and Format Preservation

RooCode utilizes the search/replace block format. Its strengths lie in its advanced search algorithm for locating the target block and its meticulous handling of code formatting during replacement.

Advanced Search Strategy: Middle-Out Fuzzy Matching

When an exact match for the search block fails, RooCode employs a ‘middle-out’ fuzzy matching approach via its MultiSearchReplaceDiffStrategy:

1. Estimate Region: Start searching near the expected location (potentially hinted by line numbers).
2. Expand Search: Search outwards from this central point.
3. Score Similarity: Use algorithms like Levenshtein distance to score the similarity between the search block and potential matches in the file.
4. Select Best Match: Choose the highest-scoring match that exceeds a defined threshold.

This strategy is effective for large files or when line numbers are slightly inaccurate, providing robustness against minor context shifts.

Emphasis on Indentation Preservation

Incorrect indentation is a common frustration with automated edits. RooCode implements a sophisticated system to preserve formatting:

1. Capture Original Indentation: Record the exact leading whitespace (spaces/tabs) of the matched lines in the original file.
2. Analyze Relative Indentation: Calculate the indentation of each line within the replacement block relative to its first line or surrounding block.
3. Apply Original Style with Relative Structure: Re-apply the captured original indentation style while maintaining the calculated relative indentation structure of the replacement code.

This detailed attention to indentation is crucial for code readability and syntactic correctness (especially in languages like Python).

RooCode Edit Process Example

<<<<<<< SEARCH
:start_line:10
-------
function calculateTotal(items) {
  return items.reduce((sum, item) => sum + item, 0);
}
=======
function calculateTotal(items) {
  // Add 10% tax
  return items.reduce((sum, item) => sum + (item * 1.1), 0);
}
>>>>>>> REPLACE

1. RooCode parses the diff:

   • Extracts the start line (10)
   • Extracts the search block (function calculateTotal...)
   • Extracts the replace block (function calculateTotal...)

2. RooCode applies the diff:

   • Reads the current content of the file
   • Uses fuzzy matching to find the best match for the search block
   • Applies the replacement with proper indentation preservation
   • Shows a diff view for user approval
   • Applies the changes if approved

3. Feedback to LLM:

   • If successful: “Changes successfully applied to file”
   • If failed: Detailed error message with the reason for failure

RooCode combines robust fuzzy matching with a strong focus on maintaining code formatting integrity.

Cursor: Specialized AI for Change Application

While other systems refine edit formats or matching algorithms, Cursor introduces a dedicated AI model specifically for the application step of the edit process.

This directly addresses the observation that even powerful LLMs, skilled at code generation and reasoning, may struggle to produce perfectly formatted, precisely located diffs that apply cleanly via simple algorithms, particularly in complex files.

Cursor’s approach involves a two-step AI process:

1. Sketching: A primary, powerful LLM generates the intended change, focusing on the core logic rather than perfect diff syntax. This might be a code block or a rough description.
2. Applying: A separate, custom-trained “Apply” model receives this sketch. This specialized model is trained to intelligently integrate the sketch into the existing codebase, handling nuances of context, structure, and potential imperfections in the input sketch. It performs more than simple text matching; it aims for intelligent code integration.

This strategy separates high-level change generation from the detailed mechanics of application. The primary LLM focuses on what to change, while the specialized Apply model focuses on how to integrate that change robustly and accurately into the file system.

You can hear the Cursor team discuss this approach:

Evolution and Convergence of Edit Formats

Examining these systems reveals interesting patterns in format development:

1. Search/Replace Lineage: Aider’s EditBlock format (<<<<<<</>>>>>>>) established an intuitive approach later adopted by Cline, which RooCode then built upon.
2. OpenAI’s Patch Influence: The specific patch format released with GPT-4.1 gained traction due to focused model training. Used natively by Codex, it was also adopted as an option by Aider.
3. Underlying Principles: Despite different origins, successful formats converge on key ideas noted by OpenAI: avoiding line numbers and clearly delimiting the original and replacement code. These features appear fundamental for reliable AI-driven editing.

Conclusion and Key Learnings

Investigating how AI coding assistants edit files reveals complex processes involving sophisticated techniques and evolving strategies.

Key Learnings:

1. Format Matters: Formats avoiding line numbers and clearly separating before/after code (like OpenAI’s patch or search/replace blocks) are prevalent and effective, especially when models are trained on them.
2. Robust Matching is Essential: Successful systems employ layered matching strategies (exact, then increasingly fuzzy) to balance precision with the ability to handle minor discrepancies.
3. Indentation Integrity is Crucial: Careful preservation of whitespace and indentation (as emphasized by RooCode) is vital for code correctness and developer acceptance.
4. Informative Feedback Enables Correction: Detailed error messages (like Aider’s) are critical for enabling the AI (or user) to diagnose and fix failed edits effectively.
5. Specialization Shows Promise: Using dedicated AI models for specific sub-tasks like change application (Cursor) represents an advanced approach to improving reliability.

Considerations for Tool Builders

Developing robust AI editing tools involves several considerations:

1. Implement Layered Matching: Start with strict matching and add fallback fuzzy strategies.
2. Prioritize Indentation Preservation: Invest effort in accurately maintaining formatting.
3. Design Actionable Error Feedback: Provide specific, informative error messages.
4. Leverage Existing Formats and Implementations: Consider established formats and study open-source systems (Aider, OpenHands, RooCode/Cline).

The Wild Horses That Weren’t: The Surprising Tale of America’s Free-Roaming Horses

Picture this: You’re gazing out over a sprawling plain as the sun sets in a blaze of oranges and purples. In the distance, a herd of horses gallops freely, manes flying like they’re in some kind of shampoo commercial. It’s wild, it’s raw, it’s… the epitome of freedom.

But here’s the twist: Those “wild” horses? Not exactly wild.

We’re about to dive into a time-traveling adventure that flips the script on everything you thought you knew about these majestic creatures roaming free across America.

Let’s rewind—way back to about 55 million years ago. North America was the original homeland of horses. Imagine a creature the size of a small dog, with multiple toes, munching on leaves in a lush forest. Meet Eohippus, the ancient ancestor of the modern horse.

Over millions of years, these early forest-dwelling horses evolved into larger, single-toed grazers adapted to open grasslands. They transitioned from nibbling on soft leaves to munching on tough grasses. North America became a vast playground of grasslands where these horses thrived.

Then, around the end of the last Ice Age, approximately 10,000 years ago, horses disappeared from North America. One moment they were everywhere, and then, over a relatively short period, they were gone.

What happened?

The end of the Pleistocene epoch brought dramatic changes to the environment. Climate shifts transformed vast grasslands into forests and tundra. Food sources dwindled, and the habitats horses relied on changed faster than they could adapt.

And then humans showed up—hungry, resourceful humans who looked at horses and saw a walking buffet. The exact reasons are still debated among scientists, but most agree that a mix of climate change and human activities played a role in the horses’ extinction on the continent.

For thousands of years after that, North America was a land without horses. Indigenous peoples built rich, complex societies without ever knowing the thunder of hooves across the plains. No galloping across the plains, no epic horse-mounted hunts—just humans and their own two feet (and sometimes canoes).

Fast forward to 1492. Columbus accidentally bumps into the Americas while looking for a shortcut to India. Over the next couple of centuries, European explorers and settlers arrive, bringing all sorts of things—some good, some bad, and some that would change the continent forever.

One of those things was the horse.

The Spanish, in particular, were big on horses. They used them for exploration, conquest, and just looking generally intimidating. These were domesticated horses, trained and bred for human use.

But horses being the free-spirited creatures they are, some managed to escape. Maybe they got spooked during a thunderstorm, or maybe they just decided they’d had enough of carrying conquistadors around. Whatever the case, these escapees started living it up in the wild, doing horse things—eating grass, making babies, and rediscovering their ancestral homeland.

These free-roaming horses came to be known as “mustangs,” a term derived from the Spanish word mestengo, meaning “stray animal.”

Now, here’s where things get really interesting.

Indigenous peoples, who had been horseless for millennia, suddenly had these new, strange creatures roaming their lands. Over time, they figured out how to catch them, tame them, and incorporate them into their daily lives. And they didn’t just stop at basic domestication—they became some of the finest horsemen the world has ever seen.

By the 17th and 18th centuries, tribes like the Comanche, Sioux, and Cheyenne had become master horsemen. The Comanche, in particular, went from being pedestrian hunters to some of the most skilled horse riders and warriors the world had ever seen—essentially inventing new forms of mounted warfare on the Plains.

They embraced the horse with ingenuity and adaptability. Horses revolutionized hunting (buffalo hunting went from being really hard to ridiculously efficient), travel (why walk when you can ride?), and warfare (now with more horsepower and mobility).

So yes, indigenous peoples like the Comanche didn’t always have horses. But once horses were reintroduced by Europeans, they adopted them quickly and integrated them deeply into their cultures, showcasing remarkable adaptability.

Alright, let’s tackle the elephant—or rather, the horse—in the room.

We often call these free-roaming horses “wild,” but technically, they’re “feral.” What’s the difference?

• Wild Horses: Horses that have never been domesticated. The Przewalski’s horse from Mongolia was long considered the only true wild horse species left.

• Feral Horses: Horses that are descended from domesticated ancestors but now live in the wild.

Interestingly, recent genetic studies suggest that Przewalski’s horses may themselves descend from early domesticated horses, blurring the lines even further. The scientific community continues to explore this, but for now, it’s clear that the mustangs of the American West are feral horses.

But let’s be honest, “feral horse” doesn’t have the same romantic ring to it. It sounds like a horse that’s going to rummage through your trash. “Feral West” sounds more like a post-apocalyptic movie than the stuff of legends.

Scientists love a good mystery. By digging up fossils, they use radiocarbon dating to figure out when these ancient horses lived and DNA analysis to understand how they’re related to modern horses. They’ve confirmed that there’s a significant gap between the ancient horses that went extinct around 10,000 years ago and the modern horses reintroduced by Europeans.

Radiocarbon Dating Explained (Without Melting Your Brain):

All living things contain carbon, and a tiny fraction of that carbon is a radioactive type called carbon-14. While an organism is alive, it keeps a steady amount of carbon-14 because it’s constantly eating, breathing, or otherwise exchanging carbon with its environment.

When the organism dies, it stops taking in new carbon. The carbon-14 it has starts to decay at a known rate—a half-life of about 5,730 years. This means that every 5,730 years, half of the carbon-14 decays into nitrogen.

Scientists can measure how much carbon-14 is left in a fossil and, knowing the rate of decay, calculate how long it’s been since the organism died. This method is effective for dating materials up to about 50,000 years old, which covers the timeframe we’re talking about.

Using this method, scientists determined that native North American horses disappeared around the end of the last Ice Age. The absence of horse fossils after that time and the lack of horse imagery in indigenous cultures before European contact support this timeline.

The reintroduction of horses didn’t just change things for the horses—it reshaped entire cultures.

For indigenous peoples, the horse was a game-changer. It altered hunting practices, made long-distance travel more feasible, and became central to warfare and status. Societies evolved rapidly, and the horse became woven into myths, stories, and identities.

Their ingenuity and adaptability in integrating the horse into their cultures is a testament to the resilience and innovation of Native American tribes.

But it’s not all majestic gallops into the sunset. Free-roaming horses have a significant ecological impact.

• Overgrazing: Horses can overgraze vegetation, leading to soil erosion and degradation of habitats.

• Competition with Native Species: They compete with native wildlife like pronghorns and bighorn sheep for food and water resources.

• Water Sources: Their presence can affect riparian areas, impacting water quality and availability for other species.

This has led to debates about how to manage their populations humanely and sustainably. While they may not be “wild” in the technical sense, they’re undeniably part of the American landscape now, and balancing their presence with environmental conservation is an ongoing challenge.

Horses in America are like that friend who moves away in elementary school and then suddenly shows up years later, totally transformed. They originated here, disappeared for a long time, and then came back under completely different circumstances.

They’re symbols of freedom and wildness, yet their very existence here is tied to human history and intervention. They represent both the untamed spirit of nature and the complex ways humans interact with the environment.

At the end of the day, the story of America’s free-roaming horses is a reminder that history is full of twists, turns, and unexpected returns. Horses have become an enduring symbol of the American West—not because they’ve always been here, but because of the incredible journey they’ve taken alongside us.

They embody resilience, adaptability, and that wild streak that runs through the heart of the American identity.

So, the next time you see a photo or painting of mustangs running free across the plains, you’ll know the real story. It’s not just a scene of wild beauty; it’s a complex tapestry of evolution, extinction, reintroduction, and cultural transformation.

And that’s way cooler than any myth.

P.S. If you ever get the chance to see these horses in the wild (or, well, the “feral”), take a moment to appreciate the epic saga they’ve been part of. They’re not just horses; they’re living history galloping across the plains.

How to become fast at decision making

When discussing the concept of decision-making in businesses, Jeff Bezos’s insights provide a profound perspective, particularly on the common pitfall many companies face: failing to recognize “two-way doors.” This failure often leads to unnecessary delays and hinders agility in the corporate world.

Most decisions in a business, according to Bezos, are “two-way doors.” These are decisions that are reversible and less critical. If a mistake is made, it’s relatively easy to backtrack and choose a different path. However, Bezos advocates that these two-way door decisions should primarily be made by single individuals or very small teams within the organization. This approach empowers teams to act swiftly and efficiently, avoiding the trap of over-deliberation.

Many companies treat these decisions as if they are “one-way doors” - significant, irreversible choices that require extensive deliberation. This cautious approach, while prudent for genuinely critical decisions, becomes a hindrance when applied indiscriminately. By applying the heavy, slow-moving process meant for one-way doors to all decisions, companies inadvertently stall their progress. They spend excessive time analyzing and deliberating choices that could be made quickly and adjusted if necessary. This not only slows down the decision-making process but also stifles innovation and responsiveness to changing market conditions.

Bezos’s philosophy at Amazon was to empower individuals and small teams to make two-way door decisions swiftly, reserving the meticulous, slower approach for the true one-way doors. This balance between caution and speed is crucial. It allows businesses to move quickly on most fronts while still being deliberate where it counts.

Getting to the truth

Tackling Groupthink in Meetings:

Groupthink is a common phenomenon in meetings, especially those involving individuals of varying seniority. Jeff Bezos sheds light on this issue, emphasizing that when a senior member expresses their opinion first, it can inadvertently influence the thoughts of others. This dynamic leads to a situation where diverse opinions may get suppressed or altered in favor of aligning with the leader’s view.

But why does this happen, even among the most competent and confident individuals? The answer lies in our inherent nature as social beings. As Bezos points out, humans are not primarily truth-seeking; we are social animals. Our survival and success have historically depended on our ability to cooperate and align with our social groups. This instinct is deeply ingrained and can subtly influence our behavior in group settings.

In a meeting, when a respected or senior figure voices their opinion, it triggers an almost instinctual response in others to align with that viewpoint. This isn’t necessarily about agreement or disagreement on a rational level; it’s about the social dynamics of respect, authority, and the desire for harmony within the group. Even highly experienced and intelligent individuals are not immune to this social influence.

To counteract this, Bezos practices speaking last in meetings. Ideally, participants state their opinions from most junior to the most senior role, ensuring that all voices are heard in an unfiltered manner. This approach not only encourages honest expression but also highlights the importance of every team member’s perspective.

The Peril of Proxies:

In business, the use of proxies – indirect measures to gauge performance or success – is common. Yet, as Jeff Bezos highlights, the management of these proxies can often lead to skewed decisions and strategies. This usually happens when organizations lose touch with the original purpose behind these proxies.

One major issue is organizational inertia. Over time, the reasons behind the selection of certain metrics as proxies can get lost in the shuffle of daily operations. Teams might continue tracking these metrics out of habit, not because they still provide relevant or useful insights. What made sense as a proxy five years ago might not be relevant today. Markets evolve, consumer behaviors shift, and what once was a reliable indicator of success or performance might now be outdated or misleading. This evolution can render once-crucial proxies ineffective, yet companies may continue to rely on them without recognizing their diminished relevance.

There’s often a lack of critical reassessment of proxies. In many organizations, questioning the validity and effectiveness of established metrics is not a regular practice. This lack of scrutiny can lead to a situation where businesses optimize for metrics that no longer align with their current goals or market realities.

To avoid the pitfalls of mismanaged proxies, Bezos suggests fostering a culture that continuously questions and reassesses these metrics. It’s vital for organizations to regularly review their proxies to ensure they still represent their true objectives and adapt to the dynamic nature of the business environment. This ensures that decision-making and strategy remain focused on actual goals, not just the numbers that are meant to represent them.

Revolutionizing Meetings with the 6-Page Memo:

Jeff Bezos’ introduction of the 6-page memo to meetings at Amazon and Blue Origin marks a significant departure from traditional corporate meeting practices. This method ensures that every participant is not just physically present but also intellectually engaged with the matter at hand.

At the core of this approach is the ‘study hall’ session, where the meeting commences with everyone silently reading a narratively structured memo for about 30 minutes. This practice counters a common problem in many companies where participants come to meetings either unprepared or having only skimmed through the pre-read materials. In such scenarios, discussions can lack depth and understanding, leading to surface-level conversations and often, misguided decisions.

Another critical issue in traditional meetings is the reliance on PowerPoint presentations, which Bezos views as a persuasion tool rather than a means for truth-seeking. Presentations with slides filled with bullet points can be misleading, allowing for vague and incomplete information to be conveyed. This method often leads to discussions that are more about aligning with the presenter’s perspective rather than delving into the actual substance of the issue.

In contrast, the 6-page memo demands comprehensive thinking and clarity from the author. This rigorous process of writing, rewriting, and editing ensures that the author presents their best thinking, leaving little room for ambiguity or half-baked ideas. For the participants, this means they are not spending time trying to extract the presenter’s thoughts during the meeting but are instead coming in with a full understanding of the subject.

Post the reading session, the meeting transforms into a dynamic discussion space, often described by Bezos as ‘messy.’ Here, the real problem-solving occurs, with participants exploring solutions based on a shared understanding developed through the memo. This method is especially effective in preventing higher-ranking individuals from unduly influencing the discussion, as everyone’s input is based on the same detailed document.

This approach also addresses the common problem of interruptions in meetings. Often in traditional settings, senior executives interject with questions, some of which would be addressed later in the presentation. The memo approach eliminates this by providing all the necessary information upfront, allowing for a more structured and focused discussion.

So what is first principles thinking and how can you become a first principles thinker?

What is First Principles Thinking

First principles comes from philosophy and physics. There a first principle is a fundamental truth that can not be broken down any further. From this fundamental truth you can reason up and explain the processes.

Elon Musk about his way of thinking at TED ¹:

Well, I do think there’s a good framework for thinking. It is physics. You know, the sort of first principles reasoning. What I mean by that is, boil things down to their fundamental truths and reason up from there, as opposed to reasoning by analogy. Through most of our life, we get through life by reasoning by analogy, which essentially means copying what other people do with slight variations. And you have to do that. Otherwise, mentally, you wouldn’t be able to get through the day. But when you want to do something new, you have to apply the physics approach. Physics is really figuring out how to discover new things that are counterintuitive, like quantum mechanics.

First Principles Thinking in Philosophy: Aristotle’s Approach

First principles thinking has its roots in the philosophical teachings of Aristotle, who used this approach to uncover fundamental truths and build a coherent understanding of the world. He called these fundamental truths “archai,” which translates to “beginnings” or “principles.” In his philosophical inquiries, Aristotle sought to understand the essence of things by identifying their fundamental building blocks and deriving knowledge from these basic truths.

To Aristotle, first principles were self-evident and indubitable truths that could not be derived from any other principles. He believed that by starting with these first principles, one could logically deduce other truths and build a solid foundation for understanding the world. Aristotle’s process of reaching these first principles involved a technique called “dialectic,” which was an exploration of different opinions and viewpoints through dialogue and questioning. By engaging in dialectic, Aristotle aimed to peel away the layers of complexity and ambiguity, ultimately revealing the foundational principles that underlie various phenomena.

One famous example of Aristotle’s application of first principles thinking is his concept of the “unmoved mover.” He reasoned that if everything in the universe is in motion, there must be a cause for this motion, and that cause must be something that is itself unmoving. By identifying the unmoved mover as a first principle, Aristotle developed a comprehensive metaphysical framework that accounted for the motion and change observed in the world.

Descartes’ Radical Doubt: “I think, therefore I am”

René Descartes, the 17th-century French philosopher, began his philosophical journey by doubting everything he knew or believed to be true. He questioned the reliability of his senses, the existence of the external world, and even the validity of his own thoughts. Through this process of relentless questioning and doubt, Descartes aimed to identify the most fundamental and self-evident truths, from which he could construct a solid and unshakable foundation for his philosophical system.

In his quest for certainty, Descartes arrived at the realization that the very act of doubting and thinking proved his own existence. He reasoned that even if he doubted everything else, he could not doubt the fact that he was doubting and thinking. This simple yet profound insight led to his famous declaration, “I think, therefore I am.”

This statement encapsulates Descartes’ first principles approach to knowledge and understanding. By questioning everything he identified a fundamental and self-evident truth: his own existence as a thinking being. From this indubitable starting point, Descartes went on to build a comprehensive philosophical system that encompassed the nature of reality, the existence of God, and the relationship between the mind and the body.

The Power of First Principles in Physics

In the late 17th century, a young and inquisitive man named Isaac Newton was studying at the University of Cambridge. A brilliant and curious student, Newton was captivated by the mysteries of the natural world and constantly sought to uncover the fundamental principles that governed it.

One day, as Newton sat beneath an apple tree in the university’s garden, he witnessed an apple falling from a branch above. This simple event sparked a profound question in his mind: what force causes objects to fall towards the ground? Inspired by this question, Newton embarked on a journey to discover the underlying principles that governed the motion of objects on Earth and in the heavens.

Through diligent research and experimentation, Newton formulated his groundbreaking laws of motion, which described the relationship between the forces acting on an object and its motion. Armed with these principles, Newton sought to apply them to the celestial realm and understand the motion of the planets and other celestial bodies.

Focusing on the Moon, Newton wondered if the same force that caused the apple to fall from the tree could also be responsible for the Moon’s orbit around the Earth. To investigate this hypothesis, he considered the gravitational force acting on the Moon and the force required to keep it in orbit.

Through a series of calculations, Newton found that the force needed to maintain the Moon’s orbit was indeed proportional to the gravitational force acting upon it, suggesting that a single force - gravity - was responsible for both the motion of the apple falling to the ground and the Moon’s orbit around the Earth. This revelation was a groundbreaking discovery that unified the understanding of forces acting on both terrestrial and celestial objects.

Newton’s insight led to the formulation of his law of universal gravitation, which stated that every object with mass attracts every other object with mass, with a force that is proportional to the product of their masses and inversely proportional to the square of the distance between them. This law provided a unified framework for understanding the motion of objects on Earth and in the heavens, forever changing the way we perceive the universe.

Through his relentless pursuit of knowledge and his ability to reason up from first principles, Isaac Newton revolutionized our understanding of the natural world. His laws of motion and law of universal gravitation continue to serve as foundational principles in physics, shaping our comprehension of the cosmos and inspiring generations of scientists and thinkers to come.

General Approach to First Principles in Physics

The first step in applying first principles thinking in physics is to formulate a hypothesis for a potential fundamental law. This often involves observing the natural world, identifying patterns, and devising a possible explanation or rule that governs the observed behavior. Physicists draw upon their knowledge of existing theories, as well as their creativity and intuition, to propose new hypotheses that can be tested through experimentation.

Once a hypothesis has been formulated, physicists design and conduct experiments to test its validity. These experiments must be carefully controlled and repeatable, allowing for the accurate measurement of relevant variables and the elimination of potential confounding factors. By comparing the experimental results with the predictions made by the hypothesis, physicists can assess whether the proposed law aligns with the observed data.

If the experimental results consistently support the hypothesis, it may become accepted as a first principle or fundamental law in physics. However, if the results contradict the hypothesis, it may need to be revised or discarded in favor of an alternative explanation.

Developing First Principles Thinking Skills

First principles thinking is a powerful mental tool that can be applied in various aspects of life, not just in physics or philosophy. By learning to think in first principles, you can develop the ability to break down complex problems, identify their fundamental truths, and reason up from there to find innovative and effective solutions. Here are some techniques to help you cultivate first principles thinking:

1. Ask “why” multiple times: When faced with a problem or question, ask “why” several times to peel back the layers of complexity and uncover the underlying principles. By questioning assumptions and diving deep into the core of the issue, you can identify the fundamental truths from which you can reason up and develop a solution.
2. Embrace doubt and skepticism: Cultivate a mindset of doubt and skepticism when tackling problems or beliefs. By questioning everything, even your own thoughts, you can identify the most fundamental and self-evident truths as your starting point. This practice enables you to build a solid foundation for understanding and solving problems, grounded in first principles thinking.
3. Challenge analogies: Many of our beliefs are based on analogies that may not be entirely accurate or relevant. To think in first principles, it’s essential to identify and challenge these analogies, scrutinizing their validity and questioning whether they truly apply in the context of the problem at hand.
4. Understand analogies and mental models: While first principles thinking encourages reasoning from fundamental truths rather than relying on analogies, it’s still valuable to be familiar with various analogies and mental models. These can serve as useful starting points for your reasoning or doubting process, helping you to identify patterns and connections between seemingly unrelated phenomena. Once you’ve drawn upon these analogies and models, you can then apply first principles thinking to refine and develop your understanding further.
5. Break down problems into their basic components: By dissecting complex issues into smaller, more manageable parts, you can analyze each component individually and identify the fundamental principles that govern them. This process will help you gain a deeper understanding of the problem and enable you to build a solution from the ground up.
6. Envision the ideal solution and work backwards: Rather than relying solely on familiar tools and methods, take a moment to imagine the perfect solution to the problem at hand. Ask yourself what the ideal outcome would look like and what characteristics it would possess. Once you have a clear vision of the desired solution, work backward to determine the necessary steps and resources to achieve it. This approach encourages you to think beyond the limitations of conventional methods, fostering creativity and innovation in your problem-solving process.
7. Embrace curiosity and continuous learning: Developing first principles thinking requires a strong sense of curiosity and a commitment to learning. By nurturing your curiosity and constantly seeking out new knowledge, you’ll be better equipped to identify fundamental truths and reason up from them to tackle complex challenges.

First Hand Examples of First Principles Thinking

Here are a few first hand examples of first principles thinking from Elon Musk. Notice how he applies the skills outlined above.

In the video below he explains how to think from first principles:

• Don’t brake the laws of physics.
• Think about how things change when you scale something to a very large or very low number. If a part is still expensive when you produce a million a year then the reason is its design.
• Anything at volume can be made for costs that asymptotically approach the costs of the raw materials plus intellectual property license rights.
• Instead of using the tools and methods that you already know ask yourself: What would be the perfect solution and how can you get to that.

Next he explains how people reason from analogy:

Batteries are expensive and they will always be, because the are expensive right now. But if you break down the material costs of batteries, then you see that these are really cheap and it is the assembly that is expensive. These costs can be reduced by improving the assembly process and increasing the scale.

On the costs of rockets. Again he breaks down the costs of a rocket to the cost of its components and the cost of assembly.

Elon about what a company and profit is:

A company is a assembly of people who gather together to create and deliver a product or service. A company has no value in itself, only in being an effective allocator to create goods and services that are of greater value than the costs of the inputs. Profit is that over time the value of the outputs is greater than the value of the inputs.

Elon about how he is able to attract talent:

If you wanna recruit people that are really talented and driven you have to state the mission and have a convincing argument for why it matters. There a three major things in terms of motivation:

• A person is enjoying the work itself intrinsically
• The financial compensation is fair and good
• The best people wanna know if what they are doing is going to matter, will people notice their work or will the world be different

Conclusion

In conclusion, first principles thinking is a powerful and transformative approach to problem-solving that has been employed by some of the most brilliant minds in history, including Aristotle, Isaac Newton, and Elon Musk. This way of thinking transcends disciplines and can be applied to various aspects of life, from philosophy and physics to business and everyday challenges. By cultivating this mindset and applying it in our own lives, we can unlock our own potential for creative problem-solving and embark on a journey of continuous growth and learning.

References

Note: The workflow described below using pyenv and venv is now largely superseded by a fantastic new tool called uv. While the principles discussed in this post remain relevant, uv offers a significantly faster and more streamlined experience.

uv acts as a replacement for pip, pip-tools, virtualenv, and venv, all rolled into one incredibly fast Rust-based package manager and resolver.

With uv, creating a virtual environment is as simple as running:

uv init

And adding packages is straightforward:

uv add <package>
# or using the familiar pip install command
uv pip install <package>

Crucially, when using uv add, it modifies your pyproject.toml to record the direct dependencies and their version constraints (similar to package.json in Node.js). uv then automatically maintains a lock file (uv.lock) which records the exact versions of all dependencies (direct and transitive) needed for your project. This lock file brings a level of dependency consistency similar to what you might be familiar with from the JavaScript ecosystem (e.g., npm and package-lock.json), something I’ve come to appreciate greatly. It ensures reproducible builds across different environments.

Given its speed and simplicity, I now recommend uv as the primary tool for managing Python virtual environments and dependencies. The rest of this post remains for historical context or if you encounter situations where uv isn’t suitable.

Original post

When I start a new Python project I use pyenv install 3.11 and pyenv shell 3.11 to install and set the Python version (here to 3.11) and then python -m venv .venv to create a virtual environment that sits in my project folder. At last it gets activated with source .venv/bin/activate. I bundled these commands into a function, so that I only need to run mkpyvenv 3.11 and the virtual environment is created and activated.

The reasons for this workflow

Every project should have its own virtual environment, so that each project is independent.

I want to be able to control the Python version for each project, eg. use 3.8 for one project and 3.11 for another.

I want my virtual environment to be in the project folder, so that when I delete the project folder, that virtual environment is also gone. Like that my system does not get littered with virtual environments that I have long forgotten about, as conda or virtualenvwrapper does.

Another benefit is, that my virtual environment does not have a name that I need to remember. I can always activate it with source .venv/bin/activate in the project folder. VSCode automatically activates a virtual environment in a .venv folder, so I don’t even have to do that.

I want to install all my dependencies with pip and a requirements.txt or pyproject.toml file (and not with conda and an environment.yml), as often a project ends up running in a Docker container. As a Docker container is its own virtual environment, I don’t want to have to install another virtual environment manager in that docker container. Also pip is the Python standard, and conda is only common in the scientific community.

The workflow in practice

This is for MacOS.

Installation

Use brew to install pyenv

brew update
brew install pyenv

Then follow the instructions of pyenv init to load pyenv when starting a shell. For the zsh shell that is:

# Load pyenv automatically by appending
# the following to
~/.zprofile (for login shells)
and ~/.zshrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

Then install the Python versions that you want to use. With pyenv versions you can see which ones are already installed.

pyenv install 3.8.16
pyenv install 3.11
...

Creating a virtual environment

Now let us assume we start a new project:

mkdir my_awesome_tool
cd my_awesome_tool

Now we set the Python version for the current shell session:

pyenv shell 3.11

When you run python --version you will see that the Python 3.11.1 version is used (or a newer patch version, as we have not been specific there). With which python you see that the python executable is in the .pyenv/shims directory. With pyenv which python you see that the python executable is stored in the pyenv directory /Users/fabian.hertwig/.pyenv/versions/3.11.1/bin/python.

So let us create a virtual environment in a .venv folder:

python -m venv .venv
source .venv/bin/activate

Now which python points to the virtual environment: /Users/fabian.hertwig/Projects/my_awesome_tool/.venv/bin/python. And again if you run python --version you will see that the Python 3.11.1 version is used. If you install a package, eg. pip install numpy then it will be stored in the .venv/lib/python3.11/site-packages/numpy directory.

Making shortcuts

To easily run through that process with just one command mkpyvenv 3.11, you can add the function below to your shell configuration file, e.g. .zshrc. Or you can use the awesome fig tool to create a dot file there which gets shared across all your fig installations.

mkpyvenv() {
    # Check if an argument was given
    if [ -z "$1" ]; then
        echo "Please specify a Python version to use, e.g. mkpyvenv 3.9.4"
        return 1
    fi
    PYTHON_VERSION=$1

    # Check if pyenv is installed
    if ! command -v pyenv > /dev/null; then
        echo "pyenv is not installed. Please install it, e.g. by running `brew install pyenv`"
        return 1
    fi

    # Install the python version if it does not exist
    pyenv install --skip-existing $PYTHON_VERSION

    # Create the virtual environment and activate it
    pyenv shell $PYTHON_VERSION
    python -m venv .venv
    pyenv shell --unset
    source .venv/bin/activate
}

• NDCG@k - Normalized Discounted Cumulative Gain at k
• MAP@k - Mean average precision at k
• Precision@k
• Recall@k
• R_cap@k - Capped Recall
• MRR@k - Mean Reciprocal Rank at k

These metrics calculate performance for a ranking task. Ranking is the more general task of search systems. Given:

• a corpus of documents or passages
• a set of queries
• a set of relevance scores, which define for each query how relevant each document is, in the simplest case by marking them with 0 or 1.

The system should retrieve the most relevant documents for each query and show them at the top rank.

Of course the system does not know the relevance scores and has to calculate them on its own. To calculate the metrics though you need them. We used a feedback system, where users could vote on search results if they are relevant for their search or not.

Example

I will use the following example to illustrate the metrics.

Corpus:

Document 1: "How to train a cat"
Document 2: "How to train a dog"
Document 3: "How to train a parrot"
Document 4: "How to train a hamster"

Queries:

Query 1: "train cat"
Query 2: "train dog"
Query 3: "train parrot"

Relevance scores:

Let us also assume, the the search system returned the documents in this order for each query.

Query 1, Document 1: 1.0 (maximum relevance)
Query 1, Document 2: 0.5
Query 1, Document 3: 0.3
Query 1, Document 4: 0.1

Query 2, Document 1: 0.7
Query 2, Document 2: 1.0 (maximum relevance)
Query 2, Document 3: 0.2
Query 2, Document 4: 0.1

Query 3, Document 1: 0.4
Query 3, Document 2: 0.2
Query 3, Document 3: 1.0 (maximum relevance)
Query 3, Document 4: 0.1

NDCG@k - Normalized Discounted Cumulative Gain at k

Normalized Discounted Cumulative Gain at k (NDCG@k) is a metric used to evaluate the performance of a ranking model. NDCG@k measures the usefulness of the top k items in the ranking, taking into account both the relevance of the items and their order in the ranking.

To calculate NDCG@k, the discounted cumulative gain (DCG) of the top k items in the ranking is first calculated. The DCG is a measure of the relevance of the items in the ranking, where higher relevance scores are given more weight. The DCG value is calculated by summing the product of the relevance of each item and a discount factor that decreases as the rank of the item increases.

Next, the maximum possible discounted cumulative gain (IDCG) of the top k items is calculated. This is the DCG of the ideal ranking, where the most relevant items appear at the top.

Finally, the NDCG@k score is calculated by dividing the DCG of the top k items by the IDCG of the top k items. The NDCG@k score ranges from 0 to 1, with a higher score indicating a more useful ranking.

Formula

NDCG@k = DCG@k / IDCG@k

Where:

• DCG@k is the discounted cumulative gain for the top k items in the ranking
• IDCG@k is the maximum possible discounted cumulative gain for the top k items

The formula for DCG@k is:

DCG@k = ∑ rel_i * (1 / log_2(i+1)) for i = 1 to k

Where:

• rel_i is the relevance of the item at rank i
• k is the number of items in the top k portion of the ranking

The formula for IDCG@k is:

IDCG@k = ∑ max_rel_i * (1 / log_2(i+1)) for i = 1 to k

Where:

• max_rel_i is the maximum relevance among all items at rank i

Example:

Here is an example of how NDCG@k can be calculated for the corpus, set of queries, and set of relevance scores from above:

Let’s say we want to calculate NDCG@2 for each query. First, we need to calculate the DCG@2 and IDCG@2 values for each query.

For Query 1:

DCG@2 = 1.0 + 0.5 * (1 / log_2(3)) = 1.0 + 0.5 * (1 / 1.585) = 1.32
IDCG@2 = 1.0 + 0.5 * (1 / log_2(3)) = 1.0 + 0.5 * (1 / 1.585) = 1.32
NDCG@2 = DCG@2 / IDCG@2 = 1.32 / 1.32 = 1.0

For Query 2:

DCG@2 = 1.0 + 0.7 * (1 / log_2(3)) = 1.0 + 0.7 * (1 / 1.585) = 1.45
IDCG@2 = 1.0 + 1.0 * (1 / log_2(3)) = 1.0 + 1.0 * (1 / 1.585) = 1.62
NDCG@2 = DCG@2 / IDCG@2 = 1.45 / 1.62 = 0.89

For Query 3:

DCG@2 = 1.0 + 0.4 * (1 / log_2(3)) = 1.0 + 0.4 * (1 / 1.585) = 1.25
IDCG@2 = 1.0 + 1.0 * (1 / log_2(3)) = 1.0 + 1.0 * (1 / 1.585) = 1.62
NDCG@2 = DCG@2 / IDCG@2 = 1.25 / 1.62 = 0.77

In this example, the ranking for Query 1 has an NDCG@2 score of 1.0, which means it is a perfect ranking. The ranking for Query 2 has an NDCG@2 score of 0.89, which means it is a good ranking but not perfect. The ranking for Query 3 has an NDCG@2 score of 0.77, which means it is a lower quality ranking.

MAP@k - Mean average precision at k

Mean Average Precision at k (MAP@k) is a metric used to evaluate the performance of a ranking model, particularly in information retrieval tasks such as search engines. It measures the average precision of the top k items in the ranking, taking into account both the relevance of the items and their order in the ranking.

Precision is a measure of the proportion of relevant items in the ranking. For example, if a ranking contains 4 items and 2 of them are relevant, the precision of the ranking is 0.5.

To calculate MAP@k, the precision of the top k items in the ranking is first calculated for each query. The precision is calculated as the number of relevant items in the top k portion of the ranking divided by k. Then, the average precision is calculated by taking the mean of the precision values across all queries.

The formula for MAP@k is:

MAP@k = 1/Q * ∑ (Precision@k of query q) for q = 1 to Q

Where:

• Q is the number of queries
• Precision@k of query q is the precision of the top k items in the ranking for query q

The MAP@k score ranges from 0 to 1, with a higher score indicating a more useful ranking.

Example

Let’s say we want to calculate MAP@2 for each query, using a relevance threshold of 0.5. First, we need to calculate the precision@2 for each query.

For Query 1:

precision@2 = (number of relevant items in top 2) / 2
= (2 relevant items) / 2
= 1.0

For Query 2:

precision@2 = (number of relevant items in top 2) / 2
= (1 relevant item) / 2
= 0.5

For Query 3:

precision@2 = (number of relevant items in top 2) / 2
= (0 relevant item) / 2
= 0.0

Then, the MAP@2 score is calculated as the mean of the precision@2 values:

MAP@2 = 1/3 * (1.0 + 0.5 + 0.0)
      = 1/3 * 1.5
      = 0.5

Precision@k

Precision@k measures the proportion of relevant items in the top k items of the ranking, relative to the total number of items in the ranking.

The Precision@k score is calculated by dividing the number of relevant items in the top k portion of the ranking by k. The Precision@k score ranges from 0 to 1, with a higher score indicating a more useful ranking.

The formula for Precision@k is:

Precision@k = (number of relevant items in top k) / k

For example, suppose we have a corpus containing 10 documents, and we want to calculate the Precision@5 score for a given query. If the top 5 documents in the ranking contain 3 relevant documents, the Precision@5 score would be calculated as follows:

Precision@5 = (number of relevant items in top 5) / 5
            = 3 / 5
            = 0.6

In this example, the Precision@5 score is 0.6, which means that 60% of the top 5 documents in the ranking are relevant.

The Precision@k metric is defined even when there are no relevant documents in the corpus, whereas the Recall@k metric is not defined in this case. This makes Precision@k a useful evaluation metric when the number of relevant documents in the corpus is small or when the relevance threshold is set very high.

Example

Let’s say we want to calculate Precision@2 for each query, using a relevance threshold of 0.5. First, we need to count the number of relevant items in the top 2 items of the ranking for each query.

For Query 1:

number of relevant items in top 2 = 1
Precision@2 = (number of relevant items in top 2) / 2
= 2 / 2
= 1.0

For Query 2:

number of relevant items in top 2 = 1
Precision@2 = (number of relevant items in top 2) / 2
= 2 / 2
= 1.0

For Query 3:

number of relevant items in top 2 = 0
Precision@2 = (number of relevant items in top 2) / 2
= 0 / 2
= 0.0

In this example, the Precision@2 scores are 0.5, 0.5, and 0.0 for Query 1, Query 2, and Query 3, respectively. This means that 50% of the top 2 documents in the ranking are relevant for Query 1 and Query 2, and none of the top 2 documents in the ranking are relevant for Query 3.

Recall@k

The Recall@k score is calculated by dividing the number of relevant items in the top k portion of the ranking by the total number of relevant items in the corpus. The Recall@k score ranges from 0 to 1, with a higher score indicating a more useful ranking.

The formula for Recall@k is:

Recall@k = (number of relevant items in top k) / (total number of relevant items)

For example, suppose we have a corpus containing 10 documents, and we want to calculate the Recall@5 score for a given query. If the top 5 documents in the ranking contain 3 relevant documents and there are a total of 5 relevant documents in the corpus, the Recall@5 score would be calculated as follows:

Recall@5 = (number of relevant items in top 5) / (total number of relevant items)
         = 3 / 5
         = 0.6

In this example, the Recall@5 score is 0.6, which means that 60% of the relevant documents in the corpus are included in the top 5 documents of the ranking.

Example

Let’s say we want to calculate Recall@2 for each query from the top, using a relevance threshold of 0.5. First, we need to count the number of relevant items in the top 2 items of the ranking for each query, and the total number of relevant items in the corpus.

For Query 1:

number of relevant items in top 2 = 1
total number of relevant items = 2
Recall@2 = (number of relevant items in top 2) / (total number of relevant items)
= 1 / 2
= 0.5

For Query 2:

number of relevant items in top 2 = 1
total number of relevant items = 1
Recall@2 = (number of relevant items in top 2) / (total number of relevant items)
= 1 / 1
= 1.0

For Query 3:

number of relevant items in top 2 = 0
total number of relevant items = 0
Recall@2 = (number of relevant items in top 2) / (total number of relevant items)
= 0 / 1
= 0

If the total number of relevant documents in the corpus is 0, the Recall@k score is not defined. This can happen when there are no relevant documents in the corpus for a given query, or when the relevance threshold is set too high such that no documents in the corpus meet the threshold.

R_cap@k - Capped Recall

The R_cap@k metric is a variant of the Recall@k metric. It measures the proportion of relevant items in the top k items of the ranking, relative to the total number of relevant items in the corpus, but caps the total number of relevant documents to k.

The formula for R_cap@k is:

R_cap@k = (number of relevant items in top k) / min(k, total number of relevant items)

Measuring Recall@k can be counterintuitive, if a high number of relevant documents (> k) are present within a dataset. For example, consider a hypothetical dataset with 500 relevant documents for a query. Retrieving all relevant documents would produce a maximum R@100 score = 0.2, which is quite low and unintuitive. To avoid this we cap the recall score (R_cap@k) at k for datasets if the number of relevant documents for a query greater than k. ¹

Example:

Suppose we have a corpus containing 10 documents, and we want to calculate the Recall@5 and R_cap@5 scores for a given query. If the top 10 documents in the ranking are ranked as follows:

Rank 1: Relevant document
Rank 2: Relevant document
Rank 3: Relevant document
Rank 4: Non-relevant document
Rank 5: Non-relevant document
Rank 6: Relevant document
Rank 7: Relevant document
Rank 8: Relevant document
Rank 9: Non-relevant document
Rank 10: Relevant document

There are a total of 7 relevant documents in the corpus, the Recall@5 and R_cap@5 scores would be calculated as follows:

Recall@5 = (number of relevant items in top 5) / (total number of relevant items)
         = 3 / 7 
         = 0.43

R_cap@5 = (number of relevant items in top 5) / min(k, total number of relevant items)
        = 3 / min(5,7)
        = 3 / 5
        = 0.60

MRR@k - Mean Reciprocal Rank at k

Mean Reciprocal Rank (MRR@k) measures the average reciprocal rank of the first relevant item in the ranking, where the reciprocal rank of an item is defined as 1/rank.

The MRR@k score is calculated by summing the reciprocal ranks of the first relevant item in the top k items of the ranking for each query, and dividing the sum by the number of queries. The MRR@k score ranges from 0 to 1, with a higher score indicating a more useful ranking.

The formula for MRR@k is:

MRR@k = sum(1/rank of first relevant item in top k) / number of queries

For example, suppose we have a corpus containing 10 documents, and we want to calculate the MRR@5 score for a set of queries. If the top 5 documents in the ranking for the first query contain the first relevant document at rank 3, and the top 5 documents in the ranking for the second query contain the first relevant document at rank 1, the MRR@5 score would be calculated as follows:

MRR@5 = (1/3 + 1/1) / 2
      = (0.33 + 1.00) / 2
      = 1.33 / 2
      = 0.67

In this example, the MRR@5 score is 0.67, which means that on average, the first relevant document in the ranking is at rank 3 for the first query and rank 1 for the second query.

Example

Let us again assume the threshold for a document being relevant is a score of greater or equal than 0.5.

• For Query 1, the first relevant document (Document 1) is at rank 1, so the reciprocal rank is 1/1 = 1.0.
• For Query 2, the first relevant document (Document 2) is at rank 1, so the reciprocal rank is 1/1 = 1.0.
• For Query 3, the first relevant document (Document 3) is at rank 3, so the reciprocal rank is 1/3 = 0.33.

The MRR@2 score is then calculated as the mean of the reciprocal ranks:

MRR@2 = (1.0 + 1.0 + 0.33) / 3
      = 2.33 / 3
      = 0.78

References

The Virtuous Circle of AI

The first time I heard of the virtuous cycle of AI was in this presentation from Andrew Ng, where he briefly describes it. Short enough so that you understand it, but too short to understand the significance of it. If you like, watch it from 14:40 to 16:40.

The circle states that a useful product will attract users. These users generate data and that data can be used to improve the product. That is a positive feedback loop and if you run that for some time, you have such valuable data, that your product evolved to be the best of its kind. It becomes very hard for others to reproduce your product or compete with you. Because if your product is so far ahead then it is hard for them to attract new users so that they create the needed data. To explain that further, let us look at some companies that implement the virtuous circle.

The Tesla Data Engine

The company that implements the circle top-notch is Tesla. The most famous one is the Tesla Autopilot¹ and there are even more circles, for example the data driven safety program². Tesla’s Autopilot is self driving car features, which allows the car to steer itself. Right now it needs constant supervision from the driver, but the vision is that the car can drive completely on its own.

Tesla builds electric cars, which is a useful product by itself to many people. To make the autonomous driving features useful from the beginning, Tesla integrated the Intel Mobile Eye system into their early cars before building their own self driving car system. That system powered lane keeping and traffic aware cruise control³. So Tesla has a useful product, which attracts users. Because there are millions of Tesla cars driving around, Tesla can effectively use the fleet to collect data about the Autopilot system. For example, they once observed the problem, that bikes attached to the back of a car get recognized as bikes traveling along the street, but should be recognized as a part of the car. So they issued commands to the fleet to collect images of bikes on cars. The fleet sends these images back to Tesla, they can then label these images correctly and use this dataset to retrain their self driving Neural Networks, e.g. improve their product. Tesla calls this system the Data Engine and presented it for the first time at the Tesla Autonomy Day, see the video below from 2:05:18 to 2:12:50. If you want to also get a good explanation of how Neural Networks work, then start at 1:52:00.

Tesla built the data feedback loop very intentionally into the cars. The Autopilot Hardware has additional computational capacity which allows Tesla to run Neural Networks in shadow mode, where they run in parallel to the networks in control, but are just observing the world or the system that is in control. Tesla also uses shadow mode to deploy triggers on the fleet, for example to send back video sequences where bikes are attached to the car or when the driver had to intervene⁴. Or they deploy the next version of a Neural Network in shadow mode and send back all the predictions that turned out to be wrong, for example when a model predicted that another car would cut into the line, but it did not.

Spotify, Netflix, YouTube, TikTok and other Recommenders

Streaming services like Spotify, Netflix, YouTube, and TikTok made the virtuous cycle their whole business model. Because their main business is not streaming media, but recommending media. If they were not able to recommend you the next good song to listen to, or series or video to watch, you would probably move to another streaming service that is able to do that. And to be able to recommend every user what they probably like, they use the data of the preferences of other users.

TikTok takes being a recommender to the extreme. TikTok is a platform where users can upload short videos and remix them with music or other user’s videos. When you open the app, you land on the #ForYou screen and a video that TikTok recommends you starts playing. The video will play on repeat until you scroll down to jump to the next video. If you double tap the screen you like the video. Over time TikTok will learn what interests you based on videos you liked previously, watched until the end or repeatedly or skipped. From time to time the app will throw in a video that is not in your interests either to challenge what they know about your interests or to get a sense of what is interesting to a broader user range⁵. So TikTok is continuously running the virtuous cycle of AI. The App is useful, because you can endlessly watch videos that entertain you and therefore it attracts more users. These users interact with videos and TikTok can learn which videos fit into which interest group and improve their recommendation engine. This improves the product as users get to see even more entertaining videos and the loop continues.

When a user opens the Netflix main page and does not find anything intriguing in 90 seconds, she will loose interest and move on to do something else⁶. If so Netflix has failed to deliver: A user wanted to watch something, but could not find anything and left. That is why Netflix personalizes the complete homepage to the user. On the page are rows of grouped videos, for example recommendations based on previously watched movies or genres. Inside the groups the videos are ranked by how interesting a video might be to the specific user⁷. Netflix even adapts the cover image of each video to the user⁸. As an image says more than a thousands words the cover image is the most important evidence for a user to decide if a movie or show might be interesting to her. Therefore the image should show which of the user interests the movie could satisfy: Is there an actor that the user likes, is there an action loaded chasing scene in it, a romantic relationship or is it about a mysterious sighting? To select a good image, Netflix sources multiple cover artworks that show different aspects of the movie. A system then learns which of these artworks is a good choice for each user by showing the different artworks to the user base and observing which image/movie/user combination led the users to select the movie and watch it. Before Netflix personalized the artworks, they simply showed the same artwork to every user. To make sure, that the new system leads to a better user experience, Netflix ran A/B test. In an A/B test the users a split up into groups. One is the control group which gets to see what the current system produces, the static artworks. Then there are also one or more experimental groups, where the users get to see the new system, the personalized artworks. Then different metrics are tracked for each group to see if the new system improves the user experience. The metrics could be the streaming hours or user retention. Netflix found that personalized artworks are a meaningful improvement and rolled it out to the whole user base.

A Butcher Running Tests on Customers

When you want to use the virtuous circle and you already have users that use your product, then you simply need to become data driven. That means inform your decisions with data, for example what features to develop or remove and in which direction to move on. Bernard Marr tells in his book Big Data in Practice⁹ the wonderful story of a butcher becoming data driven because of the threat of a supermarket chain that opened nearby. They installed cheap small sensors near the display window, sandwich board and inside the shop. These sensors are able to pickup smartphone signals and can therefore measure how many people stopped at the window, board and hopefully went into the shop. That allowed them to run tests on what to display in their window and what to write on the board and measure how that affects customer numbers. Think about that, a butcher running A/B tests! They found out that a recipe fitting the current time of year was more effective in attracting customers than a message advertising a cheap price. They also found out that a lot of foot traffic passed their shop in the late evening, when their shop was long closed, because there were two pubs nearby. Opening the shop in the late evening to sell sandwiches to the pub dwellers turned out to be a lucrative additional business.

Buffer Testing the Product Idea

Buffer is a company that allows you to prepare Twitter posts and schedule them to be posted in the future. Even before they had built the product and it was merely a good idea, they started the virtuous cycle. They created to most minimum valuable product you can imagine and tested the value of their idea.

_{Source: Idea to Paying Customers in 7 Weeks: How We Did It}

They created a landing page which described the the product in a few lines of text. Next to it was a button Plans and Pricing. When visitors clicked the button they landed on the next page that stated: Hello! You caught us before we are ready. If you’d like us to send you a reminder once we are ready put your email in below¹⁰. From the number of people that clicked the Plans and Pricing button and left their E-Mail, they got a pretty good idea of how useful the product would be to users. To be able to measure the value of their idea, they added a new second page after the Plans and Pricing button that actually showed some plans and pricing. When visitors then clicked on one of the plans, then they got to the page where they can leave their E-mail. This is probably a better example for Lean Startup than for the virtuous cycle of AI, but I wanted to include it to show with how little you can start collecting data. Here, Buffer does not even have a product or users, but just the idea that there might be a product in the future is already useful to some people. And if they click through your landing page they give you data on which you can base your decision.

Collecting Data to be able to Train Robots

To get back to a example where AI is actually the key component, let us look at AMP Robotics who started with a unique dataset. They build robots that can sort trash to revolutionize recycling. Therefore a computer vision system must be able to detect which kind of trash is on a conveyor belt so the robot arm can pick it up and throw it in the right bin.

When they started they setup a small demo conveyor belt at their lab. The CEO went dumpster diving on the weekends to find an assortment of bottles and cans for their dataset¹¹. Once the janitor even thought they threw a big party and was about to throw away all the trash that they collected, luckily he called in first. With the small dataset and the lab setup they were able to built a compelling demo and got to talk to recycling site operators. But they knew that the demo only really worked in their lab setup and would struggle under different lighting conditions and trash types. On site, they were allowed to record more video of the actual conveyor belts that transport trash which they annotated to improve their system. Once they felt ready to put a trash sorting robot arm on site they purposefully set it up somewhere, where it could not create a lot of harm, as they knew they need to collect and label more data before their robot could detect the different types of trash accurate enough to be valuable. Their robots are connected to their cloud infrastructure and send back images. These get annotated, added to the dataset and new computer vision models are trained, that the robots can then download and use. This is how AMP Robotics runs the virtuous cycle of AI and continuously improves the system over time.

Conclusion

The virtuous circle of AI: A useful product attracts more users which generate data that can be used to improve the product. Tesla uses it with their data engine to collect data from the fleet to train new versions of Autopilot Neural Networks. TikTok uses it to learn which videos are most entertaining to each user. Netflix improves the user experience by showing cover art that explain to users if a movie is interesting to them. A butcher used it to find out how to contact users in and around their shop. Buffer started without users, data or a product but tested the product idea. And at last the CEO of a robotics company goes dumpster diving to collect assets for their dataset. These were just a few examples, but I hope they showed a good range of applications and entry points.

If you want to implement to virtuous circle then think about how you could learn from users, how you can test product improvements, what is your dataset and can you improve it from user interactions.

References