“Test flights are used to gather data before flying a ‘real’ satellite and the degree of success is a function of how much data is gathered.”
Read the updateA room in the archive
Failures are data
SpaceX failed often, and in public. It wrote down each failure, its cause and its fix, and posted them for anyone to read. This room holds every failure in that record, 2003 to 2013. Seven are told in full.
64 failures and 17 stopped countdowns.
Case 1 · The test stand
2003 to 2005
Failing on purpose
What they saw
A Brief History of the Turbopump
Let me provide a brief history of our pump development and tell you where we went wrong and how we made mistakes: …
However, in September the pump started showing significant problems, including severe vibration and a few cases where the shaft itself seized up completely. What had actually happened was that we had misdiagnosed the nature of the problem, confusing cause with effect.
Read the updateThe cause
“That was the point at which we realized that we had the problem backwards.”
Read the updateThe LOX bearings overheated and melted their housings; the loose shaft shook against the seals, and the seals cut into it. The team first fixed the symptom. It took them from September 2003 to the end of the year to find the cause.
The turbopump, 2003 · Drawn by Explorist
Cause and effect, reversed
- September 2003 · Seized“However, in September the pump started showing significant problems, including severe vibration and a few cases where the shaft itself seized up completely. What had actually happened was that we had misdiagnosed the nature of the problem, confusing cause with effect.”
- May to August 2003 · What they thought“Between May and August last year, we had noticed that the seals were rubbing on the shaft and had concluded that this was overloading the bearings, resulting in vibration and occasional seizure. So the fix we put in was to increase clearance between the shaft and the seals and make up for it by increasing the amount of helium purge.” Then: “problems continued with only modest improvement.”
- What it was“That was the point at which we realized that we had the problem backwards. What was actually happening was that the bearings in the liquid oxygen were overheating (despite being -300F, liquid oxygen is a poor coolant). This caused them to melt their housings and become very loose against the shaft. Now that the shaft was not held rigidly in place, it vibrated against the seals, causing them to cut into and eventually seize the shaft.”
- December 2003 · The fix“We redesigned the turbopump to dramatically increase cooling to the LOX bearings and the problems went away. The seal design was also changed to float instead of being fixed, so even with severe vibration the seals would not cut into the shaft metal.” By December they could “run the pump at will.”
- April 2004 · The last problem“Cavitation is what occurs when the pump starts creating bubbles in the liquid as it spins … the small bubbles become a big bubble and the blades pump a combination of vapor and liquid, which causes the shaft to overspeed.” Two more designs later, the LOX pump showed “a 43% margin over the minimum design target for suction specific speed.”
The fix
More cooling to the bearings; floating seals. Then a third impeller design: “third time lucky.” By April 2004 the LOX pump had a 43% margin.
Around it, the method
The first stage welds fail SpaceX’s own inspection
A weld repair could have been attempted, but that would have been an unnecessary risk. The tank goes back to the supplier.
Read the updateA 60-second run “begins to melt the engine throat”
60-second Merlin run begins to melt the engine throat.
Read the accountTim Buzza breaks the test script on purpose. “Tim (Director of Test Operations) intentionally stepped out of sequence and unplugged a pressure transducer feed to the flight computer, to the surprise of the avionics group. The system automatically compensated with no more than a 0.1psi drop.”
Read the updateThey load the structure until it breaks. “In doing our structural qualification, we increased loads on the qualification article all the way to failure for max q-alpha, showing that we have about a 23% margin”
Read the updateMusk flies cracked engine chambers to El Segundo and spends a night smearing epoxy into them, dressed for a Christmas party. “It didn’t take long, after pressures began to rise, for the epoxy to come undone. … Musk had been wrong.” “Ablative was for sure a huge mistake,” Musk told Berger.
Berger, Liftoff, ch. 7“We can put another fucking thing on there, but I’ve blown up enough shit today.” Musk: “Okay, all right, that’s fine. Just calm down. We’ll do it again tomorrow.”
Tom Mueller, head of propulsion. Vance, Elon Musk, ch. 6, 2015. Outside, at the time
When a Merlin chamber failed in qualification in 2005, Musk told Space.com:
“To the best of my knowledge, every booster engine qualification program in history has had numerous failures of the kind that recently occurred with Merlin, so this is not unusual.”
Leonard David, Space.com, September 14, 2005. It was chamber 13; the 12 before it had worked. “We had a whole bunch of flight chambers. … And we had to hydro proof them, proof check them with pressure and they all cracked.”
Tom Mueller, vice president of propulsion. It’s Not Rocket Science, at 10:57, 2026. “There’s times when there was intentional failures. It’s like, you know, hey, let’s go try this. And like, it’s not gonna work. I don’t care. Go do it anyway.”
Chris Thompson, vice president of structures. It’s Not Rocket Science, at 13:15, 2026. “It’s not how well you avoid problems. It’s how fast you figure out what the problem is and fix it.”
Tom Mueller, head of propulsion. Isaacson, Elon Musk, ch. 18, 2023. The count
After a lot of difficult development work, several component redesigns and the occasional RUD (rapid unscheduled disassembly), the flight engine … is working.
Read the update
Merlin full-duration run. Four views of one firing on the stand in Texas. Its sound plays under this case.Video: SpaceX, Merlin full-duration run, January 2005. Watch it on YouTube Sound: SpaceX, Merlin full-duration run, January 2005. The full video plays above.
Next try
On the stand, the next try was the next day. The record gives no dates for most of these.
The cost
Crews worked ten-day stints in Texas for two years. Buzza’s one-year-old daughter said “Jeremy bad” when Jeremy Hollman came to the door, because it meant her father was leaving.
Berger, Liftoff, ch. 7The prediction
The record
“As several studies have shown, there are really two reasons why launch vehicles fail — it’s either engines or separation events.”
The record
“it is a lot better to solve problems on the ground than risk failure in flight.”
On the test stand, failure was the plan. The turbopump seized, chambers cracked, welds failed inspection. Each time the team found the cause, often after first fixing the wrong thing. Musk wrote it all up for the public, mistakes included.
Between
December 19, 2005
The pad, 2005
Is that a shadow?
On December 19, 2005 (California time), during a wind hold on Omelek, a faulty pressurization valve pulled a vacuum in the first stage fuel tank as it drained.
This caused a fuel tank barrel section to deform and suck inward. It is important to note that the root cause is an electrical fault with a valve, not structural design.
Read the updateChris Thompson saw it first on the control room screen: “Wait,” he said. “Is that a shadow?”
Berger, Liftoff, ch. 4A new stage went to the island, by air. The countdown that failed is in the record.
“You can’t really damage anything when you crash a rocket. So it’s a pretty safe place for flight testing.”
The chart marks the stopped countdowns of the Flight 1 campaign as ticks in the countdown lane.
The first launch countdown came in November 2005. Two countdowns stopped. In the second, a shorted valve pulled a vacuum in the first stage tank, and its wall caved in. Flight 1 flew four months later.
Case 2 · Flight 1
March 24, 2006
The corroded nut
What they saw
Musk’s first note, the same day: “However, at T+25s, a fuel leak of currently unknown origin caused a fire around the top of the main engine that cut into the first stage helium pneumatic system.”
Read the update“Mueller was the first to notice a problem with the Merlin engine. ‘Oh shit!’ he exclaimed.”
Berger, Liftoff, ch. 4“About thirty minutes after that accident, Elon, Chris, Tom, Hans, and I jumped into a helicopter, flew out to Omulek. It looked like a war movie. Our dreams were literally scattered everywhere.”
More than a hundred island residents met the team at the dock to help. They laid the pieces out on the hangar floor in the shape of the rocket: “One person might pop out of the water and say, ‘Hey, look, I found a turbopump!’” The satellite fell through the roof of the island’s machine shop.
Berger, Liftoff, ch. 4The cause
A fuel leak, a fire, and a nut
A fuel leak started at T-400 seconds at the turbopump inlet pressure transducer; ignition lit it; the fire burned through a pneumatic line; the valves closed; thrust ended at T+34 seconds.
After reviewing several possible, but unlikely, causes of the leak, the board concluded that the most plausible scenario was that the aluminum B-nut on the transducer failed shortly before launch due to stress corrosion cracking. … “Unfortunate” would be an understatement.
Read the findingsDays to cause: 1 to find the leak’s location; 116 to the board’s finding.
The skeptic, at the right moment: the board’s co-chair was Pete Worden. Before the flight he had written to Musk with a warning. “It was being done like a bunch of kids in Silicon Valley would do software. … it struck me that it wouldn’t work.” After the crash, Musk recommended him to investigate for the government.
Vance, Elon Musk, ch. 6The findings name him as co-chair.
Drawn by Explorist
Flight 1: 434 seconds
The record’s photo opens at liftoff.
The board: “After reviewing several possible, but unlikely, causes of the leak, the board concluded that the most plausible scenario was that the aluminum B-nut on the transducer failed shortly before launch due to stress corrosion cracking.” And: “… recovered hardware showed the lockwire that led to the B-nut was attached to the fuel line.”
Elon Musk, July 21, 2006: “The nut side had been torn off, but you could still see the lock wire. … It’s not definitive, because you don’t have the nut.”
T+34 s.“… engine thrust terminated due to closure of propellant pre-valves.” Kimbal Musk, the next day: “The rocket impacted on a dead reef about 250 ft away from the launch pad, so most of it is recoverable for analysis.”
- T−400 s. “… when the propellant pre-valves were opened, a fuel leak started on the turbopump fuel inlet pressure transducer plumbing. Data shows that this same leak was not present on the test firing that occurred on March 22.”
- About T−2 s. “… ignition of the engine in turn ignited the fuel leak, initiating the engine fire.”
- Liftoff. “Video, photo and data confirm that an engine fire started just after ignition, but prior to liftoff. The fire continued to burn during all powered flight.”
- About T+25 s. “… the engine fire burned through a pneumatic line, causing pneumatic pressure to drop.”
- T+30 s. “… the fuel and oxygen pre-valves began to close as pneumatic pressure dropped below minimum levels.”
- T+34 s. “… engine thrust terminated due to closure of propellant pre-valves.” Kimbal Musk, the next day: “The rocket impacted on a dead reef about 250 ft away from the launch pad, so most of it is recoverable for analysis.”
The fix
- Aluminum B-nuts replaced by welds or stainless steel
- Fire blankets and a nitrogen purge in the engine bay
- Metal pneumatic lines
- Three sign-offs on every job at the pad
- “More than 700 parameters are now tracked continuously,” “an order of magnitude greater than in the past.”
- A note in Musk’s words: “Ironically, this is a case where a more expensive part failed!”
Next try
361 days, to Flight 2.
The cost
Two weeks after the flight, Musk said in public that a pad processing error by “one of our most experienced pad technicians” caused the leak (SpaceNews, April 2006). The recovered fuel line still held half the nut, with the technicians’ safety wire attached. Musk said so himself in July:
So it appears that the technicians in fact did secure the nut and lock wire it.
Read the interviewThe board named corrosion as the most plausible cause.
Jeremy Hollman, who led the launch crew, confronted Musk over the blame. Afterward he stayed at SpaceX on one condition: he would never have to speak with Musk again. “I have no recollection of Hollman storming into my cubicle area,” Musk told Berger.
Berger, Liftoff, ch. 6; Isaacson, Elon Musk, ch. 23“They were our best guys, and they kind of got blamed to get an answer out to the world. That was really bad. We found out later that it was dumb luck.”
Hans Koenigsmann, who led the investigation, did not speak to his wife or anyone else for a month.
Berger, Liftoff, ch. 4Years later, with a smile, of his wife:
“… now she also says that the time gets longer every time I tell it. Maybe it was just a week at first.”
“We learned a lot of things we did wrong, and learning sometimes hurts.”
Falcon 1’s first flight lasted 34 seconds of thrust. Fuel leaked from a nut cracked by salt air, and the engine caught fire at ignition. The team laid the pieces out on a hangar floor to find the cause. The fix list changed parts, procedures and software.
Case 3 · Flight 2
March 20, 2007
Risk number eleven
What they saw
The second stage’s nozzle glows red over the curve of the Earth, then begins to swing. Steve Jurvetson’s photos from the control room: “T-2 seconds” and “That’s hot! SpaceX Falcon in space”. Photos: Steve Jurvetson, March 21, 2007, on Flickr
“I was sitting next to Elon and looked at him and said, ‘We’ve made it.’ We’re hugging and believe it’s going to make it to orbit. Then, it starts to wiggle.”
The cause
In a nutshell, the data shows that the increasing oscillation of the second stage was likely due to the slosh frequency in the liquid oxygen (LOX) tank coupling with the thrust vector control (engine steering) system. This started out as a pitch-yaw movement and then transitioned into a corkscrewing motion. For those that aren’t engineers, imagine holding a bowl of soup and moving it from side to side with small movements, until the entire soup mass is shifting dramatically.
Our simulations prior to flight had led us to believe that the control system would be able to damp out slosh, however we had not accounted for the perturbations of a contact on the stage during separation, followed by a hard slew to get back on track.
Read the updateThe review, three months later: eight anomalies, one fatal.
Drawn by Explorist · a working model
The bowl of soup
Stage swingLOX sloshFaint: the same flight, no fix
Control lost
Roll, then flame-outabout 90 sSeparationTime, compressed
Changes
The swing grew in pitch and yaw, then “induced a noticeable roll torque. This roll torque eventually overcame the 2nd stage’s roll control thrusters and centrifuged the propellants, causing flame-out of the Kestrel engine.” (SpaceX flight review, June 15, 2007.)
- The knock: “The nozzle impact during stage separation occurred due to a much higher than expected vehicle rotation rate of about 2.5 deg/sec vs. max expected of 0.5 deg/sec.”
- The steering answers: “… we had not accounted for the perturbations of a contact on the stage during separation, followed by a hard slew to get back on track.”
- “For those that aren’t engineers, imagine holding a bowl of soup and moving it from side to side with small movements, until the entire soup mass is shifting dramatically.”
- The swing grew in pitch and yaw, then “induced a noticeable roll torque. This roll torque eventually overcame the 2nd stage’s roll control thrusters and centrifuged the propellants, causing flame-out of the Kestrel engine.” (SpaceX flight review, June 15, 2007.)
- The fix: “adding baffles to our 2nd stage LOX tank and adjusting the control logic.” “Either approach separately would do the trick.”
The callback
The record
“If the second stage engine nozzle does hit the interstage on exit, the worst that will happen is that the niobium nozzle will be dented”
The record
“There was no apparent damage to the nozzle, which is not a big surprise given that niobium is tough stuff.”
The nozzle survived, as designed. The knock set off the slosh.
The risk they chose. Steve Davis had run the slosh models; in a tiny share of runs, the stage spun out. Slosh sat eleventh on the list of top risks. Baffles add mass. “It wasn’t even in the top ten risks,” Musk said. “How bad could it be?” Afterward: “Now, I ask for the eleven top risks. Always go to eleven.”
Berger, Liftoff, ch. 6The fix
“Extensive 2nd stage slosh baffles will be included in all future flights, as is currently the case with the 1st stage.”
And new control logic. Days to cause: 7.
Read the reviewNext try
501 days, to Flight 3. Most of that wait was the new Merlin 1C engine (the Flight 3 summary).
The cost, and the argument
There seems to be a lot of confusion in the media about what constitutes a success. … This is not ‘spin’ or some clever marketing trick.
And he set the bar for every later flight: “Let me be clear here and now that anything less than orbit for that flight or any Falcon 1 mission with an operational satellite will unequivocally be considered a failure.”
Read the updateOutside, at the time
The press headlines read:
“Falcon flies to space but misses orbit”
Outside, at the time
The customer was calmer. An Air Force official wrote that the service
“… notes with interest their continuing effort and improvements over the last launch attempt. … We appreciate that new launch capabilities take time and effort to fully develop.”
Flight 2 reached space and fell short of orbit. Slosh in the oxygen tank shook the second stage until the stage began to roll and its engine flamed out. The team had known the risk. It was number eleven on a list of ten.
Case 4 · Flight 3
August 2, 2008
10 psi
This case tells Flight 3 as a failure. The Hail Mary room tells what came after.
What they saw
“But then, before the first stage had dropped more than a few feet, it snapped back upward. To the horror of those watching, it thudded into the bottom of the second stage.”
Berger, Liftoff, ch. 8The cause
Falcon 1, Flight 3 Mission Summary
Four methods of analysis — vehicle inertial measurement, chamber pressure, onboard video and a simple physics free body calculation — all give the same answer.
Unlike the ablative engine, the regen engine had unburned fuel in the cooling channels and manifold that combined with a small amount of residual oxygen to produce a small thrust that was just enough to overcome the stage separation pusher impulse.
Read the summary
Days to cause: 4.
Outside, at the time
Gwynne Shotwell, ten days later, told an audience how an assistant took the news:
“You mean to tell me we rear-ended ourselves?”
Mueller first blamed the separation system. Chris Thompson, who owned it, told him to look at the data first. Steve Davis worked through the night, frame by frame, and found the first stage still accelerating after it let go.
Berger, Liftoff, ch. 8Why the test stand missed it
Unfortunately, the engine chamber pressure is so low for this transient thrust — only about 10 psi — that it barely registered on our ground test stand in Texas where ambient pressure is 14.5 psi. However, in vacuum that 10 psi chamber pressure produced enough thrust to cause the first stage to recontact the second stage.
Read the summaryDrawn by Explorist
Why the test stand missed it
Chamber
10 psi
The leftover burn
Air outside
0 psi
The tick, 1 psi, cancels the push
Changes
- On the test stand in Texas the air outside presses back at 14.5 psi. The new engine’s leftover thrust made “only about 10 psi” in the chamber, so “it barely registered.” SpaceX, August 6, 2008
- In space nothing presses on the nozzle. The 10 psi in the chamber pushes the first stage forward.
- “However, in vacuum that 10 psi chamber pressure produced enough thrust to cause the first stage to recontact the second stage.” It was “just enough to overcome the stage separation pusher impulse.” SpaceX, August 6, 2008
- The fix: wait longer, so the push dies away before the stages part. The summary: “As it turned out, a very small increase in the time between commanding main engine shutdown and stage separation would have been enough to save the mission.” SpaceX, August 6, 2008
“On the test stand, we didn’t notice anything. We thought it was fine.”
“That’s what we learned from Falcon 1, that basically any difference between how you fly the vehicle and how you test it on the ground is where you’re going to have problems. That’s probably the biggest lesson we learned.”
The fix
More time between engine shutdown and separation. “As it turned out, a very small increase in the time between commanding main engine shutdown and stage separation would have been enough to save the mission.”
Read the summary“Between the third and the fourth flight we changed one number, nothing else. That was the time we needed to separate the two stages.”
Next try
57 days, to Flight 4. The shortest gap of the Falcon 1 years. The money was nearly gone.
The cost, and the line that points ahead
Good Things About This Flight
We discovered this transient problem on Falcon 1 rather than Falcon 9
Read the summaryOutside, at the time
Two weeks later he said it out loud:
“Better to learn it on our small rocket, the Falcon 1, than on our big rocket, the Falcon 9.”
“That one hurt almost as bad as Flight One, because it was so preventable.”
The new engine kept pushing for a second after shutdown. On the test stand, air pressure masked the push. In space it shoved the empty first stage into the second. The fix was a short wait before separating.
Between
Three for three
The Falcon 1 scorecard
Three flights, three causes, set under Musk’s 2004 line: “it’s either engines or separation events.”
| Flight | Where it went wrong | Cause | Days to cause | Days to next try |
|---|---|---|---|---|
| 1, Mar 2006 | Engine | Fuel leak from a corroded nut; fire | 1 (where), 116 (board) | 361 |
| 2, Mar 2007 | Separation, then the second stage | A knock at separation set off LOX slosh | 7 | 501 |
| 3, Aug 2008 | Separation | Engine thrust after shutdown | 4 | 57 |
The three were
“… what I would call design [failures]. We made an assumption and designed it that way and the assumption was wrong … we partly wanted to save weight.”
Outside, at the time
The skeptic was Musk himself. Three days after Flight 1 he told Nature:
“If we have three consecutive failures, then we probably don’t know what we’re doing.”
Wired
August 5, 2008
Now 0-for-3, SpaceX’s Elon Musk Vows to Make Orbit
After Flight 3, Wired put it back to him, with John Pike of GlobalSecurity.org saying that launching rockets is hard because of “physics, not some new design or economic model.”
Musk: That was the dumbest thing I’ve ever said. … the first order of business is to get rid of design errors, which we’re doing.
Read the whole articleFlight 4 reached orbit on September 28, 2008. That story is the Hail Mary room.
All three failures began where Musk had said rockets fail: at an engine, and at separation. Each cause was different, and none came back. In 2006 Musk had said three failures would mean they did not know what they were doing.
Between
Built to fail
The turn to Falcon 9. The record argued for failure before it happened.
As far as reliability, one could reasonably argue that the ability of F-V to lose an engine at any stage of flight and still complete a successful mission is the biggest launch vehicle reliability improvement in decades.
Read the updateOn Falcon 9’s nine engines: “If there is an engine fire or a chamber comes apart, will it be limited to that one engine or will it cause a neighboring engine to fail?”
Read the update
By late 2008 SpaceX had fired engines “more than 2,000” times in five years. Early Merlin 1C ignitions worked about a quarter of the time.
Berger, Reentry, ch. 1“It was always like, man, how are we going to start nine of these things when we have trouble starting one?”
The chart here shows the gap: 2008 to 2010, the flight lane empty while Falcon 9 is built and tested.
Los Angeles Times
June 5, 2010
Start-up shows it has right stuff
On Falcon 9’s first flight the paper quoted the White House science adviser, John Holdren, who had hedged before the result: “Failures are a relatively common occurrence in the early flights of a new rocket, and rocket scientists often learn as much from these early failures as they do from their successes.”
Read the whole articleFalcon 9 carried nine engines so it could lose one. Musk had argued for engine-out since 2003. A rocket that expects a failure must also keep that failure from spreading.
Case 5 · The cracked nozzle
December 2010
Two days
What they saw
Days before Falcon 9’s second launch, with Dragon on top, a camera inspection found cracks in the second stage nozzle extension.
SpaceX engineers are analyzing two small cracks in the aft end of the 2nd stage engine nozzle extension.
Read the updateA new nozzle would take about a month.
Berger, Reentry, ch. 4The cause
A nitrogen vent line blew on the thin end of the nozzle until it fluttered and cracked. Spaceflight Now reported it on launch day.
NASA had suggested the purge, to stop ice; at the launch readiness review the team “played the video showing the purge vent blowing nitrogen onto the nozzle, which fluttered like a paper in the wind.”
Berger, Reentry, ch. 4Days to cause: 2.
The fix
Cut off the cracked end. The record already named it: “The most likely path forward is that we will trim off the thinnest portion of the nozzle extension”
Read the updateTechnician Marty Anderson flew in with 30 pairs of tin snips and cut 14 inches from the bottom of the nozzle, about 25 feet around, in four hours. His snips wore through his glove to the bone. He asked for duct tape and finished.
Berger, Reentry, ch. 4Drawn by Explorist
The cracked nozzle, 2010
The bell at the bottom of Falcon 9’s second stage engine. In SpaceX’s words, it “is made of niobium sheet alloy, measures 9 feet tall and 8 feet at the base diameter, and thins out to about twice the thickness of a soda can at the end.”
- The nozzle. The bell at the bottom of Falcon 9’s second stage engine. In SpaceX’s words, it “is made of niobium sheet alloy, measures 9 feet tall and 8 feet at the base diameter, and thins out to about twice the thickness of a soda can at the end.”
- The cracks. December 6, 2010, two days before launch: “SpaceX engineers are analyzing two small cracks in the aft end of the 2nd stage engine nozzle extension.” They sat “in a region near the end of the nozzle extension where there is very little stress.”
- The vent line. The nozzle hung inside the interstage, between the stages. “NASA had been concerned about ice building up on top of the first stage propellant tank and had suggested SpaceX feed nitrogen into the interstage to prevent this. But the one-inch duct had then blown nitrogen gas almost directly onto the second stage nozzle” (Berger). SpaceX, December 8: “A GN2 vent line caused fluttering of the thinnest portion of the nozzle extension, creating the cracks.” GN2 is nitrogen gas.
- The cut. SpaceX: “The most likely path forward is that we will trim off the thinnest portion of the nozzle extension, which is where the cracks are located.” Technician Marty Anderson cut 14 inches off the bottom with tin snips, all the way around, in about four hours. Then the cause: SpaceX “corrected the root cause by diffusing the vent” (December 8, 2010). Falcon 9 launched Dragon that day.
- Cut from the bottom
- 14 in.
- Cut around the engine
- 25 ft
- Tin snips
- 30 pairs
- Time to cut, about
- 4 hours
- Update to launch
- 2 days
Next try
2 days. Falcon 9 launched Dragon on December 8, 2010. Dragon came home from orbit.
The skeptic, at the right moment
NASA’s own. “I’d been through a lot of stuff on the space shuttle,” said Doug Cooke, NASA’s head of exploration. “And this just seemed a little too out of the ordinary.” … “This was not our vehicle. So it was not really our choice.” Alan Lindenmoyer, who ran NASA’s commercial cargo program, had assumed weeks of delay.
Berger, Reentry, ch. 4Cracks in the second stage nozzle, found days before launch. A new nozzle would take a month. A technician cut 14 inches off the bottom with tin snips. NASA’s officials doubted it. Dragon flew two days later.
Case 6 · CRS-1
October 7, 2012
One engine out
What they saw
SpaceX CRS-1 Mission Update
Approximately one minute and 19 seconds into last night’s launch, the Falcon 9 rocket detected an anomaly on one first stage engine. Initial data suggests that one of the rocket’s nine Merlin engines, Engine 1, lost pressure suddenly and an engine shutdown command was issued. We know the engine did not explode, because we continued to receive data from it.
Read the updateThe next day: “SpaceX rocket loses 1 of 9 engines,” the Los Angeles Times.
What the design did
Falcon 9 did exactly what it was designed to do. Like the Saturn V (which experienced engine loss on two flights) and modern airliners, Falcon 9 is designed to handle an engine out situation and still complete its mission.
Read the updateThe 2005 question, answered seven years later.
“It proved Elon’s … initial claims that he had designed this rocket with an engine-out capability. They literally proved that in flight.”
A poster went up on a wall at SpaceX:
“Lost an engine and still got to Space Station 30 minutes early.”
Dragon berthed on October 10.
What it did not save
A small Orbcomm satellite rode as a second payload. It needed a second burn of the upper stage. NASA’s rule for the station mission allowed that burn only with very high odds of success, and with one engine lost, the odds fell short. The satellite was left in a low orbit and burned up.
Outside, at the time
SpaceX, three days later:
“For the protection of the space station mission, NASA had required that a restart of the upper stage only occur if there was a very high probability (over 99 percent) of fully completing the second burn.”
Outside, at the time
The statement went on:
“While there was sufficient fuel on board to do so, the liquid oxygen on board was only enough to achieve a roughly 95 percent likelihood of completing the second burn, so Falcon 9 did not attempt a restart.”
Outside, at the time
Orbcomm’s release:
“The Company has filed a notice of claim under its launch insurance policy for a total loss of the OG2 prototype. The maximum amount covered by the policy is $10 million.”
Drawn by Explorist
Nine engines, then eight
Falcon 9 first stage, seen from below
“Falcon 9 shuts down two of its engines to limit acceleration to 5 g’s even on a fully nominal flight. The rocket could therefore have lost another engine and still completed its mission.” SpaceX, October 8, 2012
- Liftoff from Cape Canaveral, 8:35 p.m. ET, October 7, 2012. All nine Merlin engines on the first stage burn.
- “Approximately one minute and 19 seconds into last night’s launch, the Falcon 9 rocket detected an anomaly on one first stage engine. Initial data suggests that one of the rocket’s nine Merlin engines, Engine 1, lost pressure suddenly and an engine shutdown command was issued.” SpaceX, October 8, 2012
- “We know the engine did not explode, because we continued to receive data from it. Panels designed to relieve pressure within the engine bay were ejected to protect the stage and other engines.” SpaceX, October 8, 2012
- “As designed, the flight computer then recomputed a new ascent profile in real time to ensure Dragon’s entry into orbit for subsequent rendezvous and berthing with the ISS.” SpaceX, October 8, 2012
- “Falcon 9 shuts down two of its engines to limit acceleration to 5 g’s even on a fully nominal flight. The rocket could therefore have lost another engine and still completed its mission.” SpaceX, October 8, 2012
The second burn
A small Orbcomm satellite rode along. To reach its orbit, the upper stage had to fire a second time.
“NASA had required that a restart of the upper stage only occur if there was a very high probability (over 99 percent) of fully completing the second burn. … While there was sufficient fuel on board to do so, the liquid oxygen on board was only enough to achieve a roughly 95 percent likelihood of completing the second burn, so Falcon 9 did not attempt a restart.”
SpaceX, October 10, 2012, in Spaceflight Now
No second burn. The Orbcomm satellite was left in a low orbit. By October 11 it had re-entered and burned up.
The cause
Not in the record from the time. Three weeks later SpaceX was still working “to determine root cause” (SpaceX, October 28, 2012).
After about two months, Koenigsmann and Kevin Miller were fairly sure the engine failed “due to a poorly manufactured thrust chamber.”
Berger, Reentry, ch. 6Outside, at the time
In December Gwynne Shotwell said SpaceX had found it, but would not say yet:
“We’re not going to release what we found but I think we’ve got a good most probable cause identified. The data supports that.”
Outside, at the time
On February 28, 2013, the day before the next launch, Gwynne Shotwell named it:
“There was a material flaw that went undetected in the jacket of the Merlin engine, resulting in a breach … causing depressurization of the combustion chamber, then the flight computer recognized that depressurization and commanded shutdown.”
Days to cause: 1 to “lost pressure”; 144 to the public cause.
The fix
Every thrust chamber inspected by shearography for the next flight; then the new Merlin 1D, with a brazed chamber that “killed the problem entirely.” Falcon 9 v1.1 also put its engines in an octagon of separate bays, “protecting the other engines in case of one’s failure.”
Berger, Reentry, ch. 6Outside, at the time
Shotwell said at the time:
“We did extensive analysis to understand the problem, [and] extensive assessment and testing on these particular engines. … The field of science that we’re talking about is called NDE - non-destructive evaluation. It’s as much an art as a science, and we’re certainly getting much better at it as we mature.”
Next try
145 days, to CRS-2.
The cost
“The engine issue stopped us dead in the tracks with NASA,” Koenigsmann said. SpaceX stood down for nearly five months.
Berger, Reentry, ch. 6On why SpaceX would not go public: “We would also get beaten up every time there was an anomaly on the rocket or spacecraft, as occurred on flight 4 with the engine failure and flight 5 with the Dragon prevalves.”
Read the emailOutside, at the time
The skeptics, the next day:
“… an engine failure and a less-than-nominal satellite deployment suggest that the company has some technical issues to resolve for future flights.”
Outside, at the time
“Any problem serious enough to generate debris typically results in a loss of mission.”
Outside, at the time
The customer:
“SpaceX has been a supportive partner, and we are highly confident in their team and technology.”
One of nine engines quit 79 seconds into the flight. The other eight carried Dragon to orbit, and it reached the station. A small satellite riding along was lost. The engine failure SpaceX had designed for since 2003 had come.
Between
March 1, 2013
CRS-2
One short stop. After Dragon separated, its thrusters would not all start.
After Dragon separated from Falcon 9’s second stage approximately nine minutes after launch, a minor issue with some of Dragon’s oxidizer tanks was detected. Within a few hours, SpaceX engineers had identified and corrected the issue, normalizing the oxidizer pressure and returning operations to normal.
Read the release
Dragon reached the station a day late (the arrival). Musk later named the part: “the Dragon prevalves” (his email of June 7, 2013).
Outside, at the time
Musk on Twitter, the same day:
“Issue with Dragon thruster pods. System inhibiting three of four from initializing.”
Outside, at the time
Then, to reporters:
“We’ve been able to free that up by cycling the valves, essentially pressure hammering the valves, to get that to loosen.”
Outside, at the time
“Dragon’s back-to-back flight problems will likely focus renewed scrutiny on the burgeoning private spaceflight company.”
Outside, at the time
Five weeks later Musk named the part and the cause:
“The problem was a very tiny change to the check valves that serve the oxidizer tanks on Dragon.” “The supplier had made mistakes that we didn’t catch.” The fix was new software, sent up in flight, to build pressure behind the valves and release it: “For the spacecraft you could call it kind of a Heimlich maneuver.”
Three of Dragon’s four thruster pods would not start after launch. Stuck valves from a supplier blocked the oxidizer. Engineers wrote new software, sent it up, and kicked the valves free in a few hours. Dragon reached the station one day late.
Case 7 · CASSIOPE
September 29, 2013
The rocket as a test stand
What they saw
The first flight of the larger Falcon 9 v1.1. No footage of either test was published in 2013. It put its satellite in orbit. Then two tests that no customer had paid for. The upper stage tried to restart its engine, and could not. And the spent first stage, its job done, fired its engines again to slow down over the ocean.
The cause, the upper stage
Its igniter lines froze (the finding). Days to cause: 53. The fix: insulation, and the cold oxygen kept off the lines.
Outside, at the time
SpaceX, November 21:
“The igniter fluid lines froze after longer exposure to cold oxygen. This never happened on the ground because ambient air kept the lines warm. We’ve added insulation and made sure that cold oxygen can’t impinge on the lines.”
The echo: the ground had hidden a fault again. In 2008 the air’s pressure masked the thrust; in 2013 the air’s warmth masked the freezing.
The first stage
“This time, there was no fireball. Instead, the rocket successfully fired three engines to slow its descent through the atmosphere. While the landing burn did not occur entirely as intended, the rocket still made it down to the surface of the ocean in one piece before corkscrewing and slamming into the water.”
Berger, Reentry, ch. 7The stage had started to spin, and the spin threw the fuel away from the engine’s inlet.
Outside, at the time
Musk, on Twitter that day:
“Rocket booster relit twice … but spun up due to aero torque, so fuel centrifuged and we flamed out.”
Outside, at the time
Before the launch SpaceX had said it plainly:
“SpaceX does not expect success with this first test.”
This was a failure chosen in advance.
Drawn by Explorist
The spin
The first stage’s fuel tank, cut away
- No spin. The leftover fuel lies over the inlet at the bottom of the tank, and the engine burns. The first relight “went well,” Spaceflight Now reported.
- The stage spins. The fuel climbs the walls, and its surface takes the shape of a bowl. The center drops toward the inlet.
- The center of the bowl reaches the bottom. Gas reaches the engine, and the flame goes out. Musk, that day: “… spun up due to aero torque, so fuel centrifuged and we flamed out.”
The stage’s way down
“SpaceX does not expect success with this first test.” SpaceX, before the launch
“I remember watching the live video and seeing the light of the engine on the ocean. … It worked the first time. I was like, get the barge ready. Get the landing legs ready.”
Next try
65 days, to SES-8, launched on December 3, 2013. Its upper stage had to restart to reach the satellite’s transfer orbit, and it did. The first stage would try its descent again in 2014, after this room ends.
The cost
Three weeks before this launch, at Vandenberg, a trainee closed six deluge valves too fast. The water main burst and flooded the pad. The test engineer offered to resign.
Berger, Reentry, ch. 6“We did not hire you for everything to go right. We hired you for when things are messed up.” Zach Dunn to the engineer who offered to resign.
Berger, Reentry, ch. 6By 2013 SpaceX used its own rocket as a test stand. After it let go of the second stage, the spent first stage relit its engines to slow down over the ocean. It spun and fell, but it reached the water in one piece. The data shaped the landings to come.
The exit
The ledger
| What happened | Cause | Known from | |||||
|---|---|---|---|---|---|---|---|
| 2003 to 2005 (dates not stated) | Test stand | Merlin development engines | “the occasional RUD”; “several” RUD eventsOpen the recordCase 1 | not stated | The record | ||
| 2003 | Test standTest stand, McGregor | Test stand tank | Lightning cut the tank’s pressurization; the tank bulgedOpen the timeline | Lightning | Later accounts | ||
| 2003 to 2004 (dates not stated) | Test standTest stand, McGregor | Merlin development engines; a test stand | A stuck fuel valve blew up an engine; another test burned down a stand; two blow-ups in one dayOpen the timeline | Stuck valve; others not stated | Later accounts | ||
| by 2003-04 | Test standTest stand, Texas | Merlin ablative chamber: insert that joins injector to chamber | “an issue with the metal insert” delays the ablative firingOpen the record | not stated | The record | ||
| 2003-07 (late) | Test standTest stand, Texas | Kestrel upper stage engine | “initially disappointing results” on first firingsOpen the record | not stated | 230 | The record | |
| 2003-09 | Test stand | Merlin turbopump | Severe vibration; shaft seized (note 3)Open the recordCase 1 | LOX bearings overheated and melted their housings; the loose shaft shook against the seals, and the seals cut into it. First misread as seal rub | 122 | 91 | The record |
| 2003-10 | Test standTest stand, Texas | Merlin thrust chamber, 60 s run | “the exhaust began to melt the metal in the engine’s throat”Open the recordCase 1 | Heat; seals also at risk. No statement from the time | 2160 | The record | |
| 2003 (late) to 2004 | Test standTest stand and factory | Merlin ablative thrust chambers | “With chamber after chamber, the ablative coating would bubble and then crack.” Musk’s epoxy repair failed in the first pressure testOpen the timelineCase 1 | Ablative design; not stated for the exact flaw | Later accounts | ||
| 2003-12 | GroundGround, Texas | Falcon 1 first stage tank welds | Welds “below our quality requirements”Open the recordCase 1 | Supplier welds below SpaceX quality | 0 | 78 | The record |
| 2003-12 | Test stand | Merlin turbopump LOX impeller | “The only showstopper”: low cavitation marginOpen the recordCase 1 | Low suction specific speed on the LOX impeller | 0 | 122 | The record |
| by 2004-07 | Test stand | Merlin fuel manifold, cast aluminum | “the old design cracked during testing”Open the record | Cast aluminum cracked; no deeper cause given | The record | ||
| 2005-01-25 | Test standTest stand, McGregor | Falcon 1 first stage (the Flight 1 stage) | Split in half at 110% of launch pressureOpen the timeline | Welds (Berger) | Later accounts | ||
| 2005-05 (before 05-27) | CountdownPad, Vandenberg | Falcon 1 static fire | AbortOpen the record | “an igniter sensor failure”; the igniter worked | 0 | 26 | The record |
| 2005-05 (before 05-27) | CountdownPad, Vandenberg | Falcon 1 static fire | AbortOpen the timeline | “an incorrectly closed helium ground supply valve” | 0 | The record | |
| 2005-09 (by 09-14) | Test standTest stand, Texas (qualification) | Merlin ablative chamber 13 | An ablative chamber “fail[ed] during the Merlin qualification program”Open the timelineCase 1 | Outside: “a variance in manufacturing, given that the prior 12 worked fine” (Musk) | 0 | 73 | Press of the time |
| 2005-11 | GroundGround, Omelek | Omelek LOX plant; LOX supply | Plant “broke down”; 20% of the Hawaii LOX arrivedOpen the recordBetween cases | Plant: not stated. Supply: poor containers | 11 | 23 | The record |
| 2005-11 to 2006-03 (dates not stated) | CountdownPad, Omelek | Falcon 1 autosequence | 3 abortsOpen the recordBetween cases | Video camera glitch; pad power surge; ground helium line came off early | The record | ||
| 2005-11-26 | CountdownPad, countdown | Falcon 1 engine computer | Engine computer reset (“reboot anomaly”)Open the recordBetween cases | Ground power: long umbilical and heat pulled voltage below minimum | 11 | 23 | The record |
| 2005-11-26 | CountdownPad, Omelek | Falcon 1, Flight 1 | ScrubOpen the recordBetween cases | A manual LOX vent valve left open; LOX boiled off; helium warmed and vented. See #9, #10 | 0 | 23 | The record |
| 2005-12-19 | GroundPad, wind hold | Falcon 1 first stage fuel tank | A barrel section deformed and sucked inward during drainOpen the recordBetween cases | Faulty pressurization valve gave a vacuum. Root: a shorted part rebooted the valve controller board | 0 | 52 | The record |
| 2005-12-19 | CountdownPad, Omelek | Falcon 1, Flight 1 | ScrubOpen the recordBetween cases | Tank damage: see #11 | 0 | The record | |
| 2006-02 | GroundPad, Omelek | Second stage avionics power box | Rocket would not power up; burned capacitorsOpen the timeline | Capacitors not rated for the raised ground voltage | Later accounts | ||
| 2006-02-09 | GroundPad, static fire count | Falcon 1 second stage tank | Pressure “acting incorrectly”, then “a small leak”Open the record | not stated. Procedures changed “to prevent such leaks” | 15 | 40 | The record |
| 2006-02-09 | CountdownPad, Omelek | Falcon 1 static fire count | The sequence stoppedOpen the recordBetween cases | Igniter timing (Kimbal) | 0 | 1 | The record |
| before 2006-03-24 | Ground | Small check valve; pressure transducer | “a few small components break”Open the recordCase 2 | Stress corrosion cracking (“apparent”) | The record | ||
| 2006-03-24 | FlightFlight, T+0 to T+34 s | Falcon 1, Flight 1 | Fire at the top of the engine; engine shut down; vehicle fell on a reef near the padOpen the recordCase 2 | Fuel leak at the turbopump inlet pressure transducer. Probable: aluminum B-nut cracked by stress corrosion | 116 | 361 | The record |
| 2006-03-24 | FlightPad, then flight | Flight 1 second stage LOX vent valve | Valve opened in tanking and never closedOpen the timeline | Procedure; no valve position sensor | Later accounts | ||
| 2007-01 (by 01-19) | GroundPad, checkout | Falcon 1 second stage TVC pitch actuator | “an anomaly with the thrust vector control (TVC) pitch actuator”Open the record | not stated (never stated) | 59 | The record | |
| 2007 to 2008 | Test stand | Merlin 1C regenerative engine | Not in the record. The record says only that the gap before Flight 3 “was mainly due to the Merlin 1C regen engine development”Open the record | not stated | Later accounts | ||
| 2007 | Test standTest stand, McGregor | Merlin 1C pintle injector | The engine “would regularly spit out” its pintle into the fieldOpen the timeline | Joint between copper pintle and aluminum injector | Later accounts | ||
| 2007-03-19 | CountdownPad, Omelek | Falcon 1, Flight 2 | ScrubOpen the record | Ground software aborted during a telemetry switch; the simulator lacked the hardware delay | 1 | 1 | The record |
| 2007-03-21 | FlightFlight, T+265 to 474 s | Falcon 1, Flight 2: second stage control | Growing oscillation, then corkscrew; engine flame-out; 289 km, 5.1 km/sOpen the recordCase 3 | LOX slosh coupled with engine steering, set off by the separation contact (#17) and a hard slew | 7 | 501 | The record |
| 2007-03-21 | FlightFlight, MECO | Flight 2: stage separation | First stage struck the Kestrel nozzle; “no apparent damage”Open the recordCase 3 | Rotation 2.5 deg/s, 0.5 expected. 2007-03-27: Merlin shutdown transient. Review: mostly air loads at a low separation (note 17) | 6 | 501 | The record |
| 2007-03-21 | FlightFlight, liftoff | Flight 2: second stage LOX quick disconnect | QD “Failed to Disconnect”; panel and ~2 in of fill line tore offOpen the record | QD design (vendor) | 86 | 501 | The record |
| 2007-03-21 | FlightFlight, liftoff | Flight 2: first stage LOX, fuel and electrical QDs | Poor disconnect; electrical lanyard brokeOpen the record | not stated | 86 | 501 | The record |
| 2007-03-21 | FlightFlight, first stage | Flight 2: first stage trajectory | Thrust low early; path “lower and slower”; LOX pump cavitated (benign)Open the record | Wrong propellant utilization file loaded; LOX tank pressure fell late | 86 | 501 | The record |
| 2007-03-21 | Flight | Flight 2: second stage propellant utilization | PU “did not operate correctly”Open the record | “still under investigation”: not stated | 501 | The record | |
| 2007-03-21 | FlightFlight, fairing jettison | Flight 2: fairing Marmon clamp band | Band halves “appear to be joined” as it fell awayOpen the record | “still under investigation”: not stated | 501 | The record | |
| 2007-03-21 | FlightOcean | Flight 2: first stage recovery | No sign of the stage; no signalsOpen the record | Range gave a wrong landing point; stage GPS locator not working | 86 | The record | |
| 2007-03-21 | CountdownPad, Omelek | Falcon 1, Flight 2 | Abort at T-0.5 sOpen the record | Chamber pressure ~0.5% low; fuel cold from the day before | 86 | The record | |
| 2007-04 | Test standTest stand, McGregor | Merlin 1C nozzles | Clogged channels and cracked parts; the last nozzle failed short of full duration on 2007-04-28Open the timeline | not stated | Later accounts | ||
| 2007 (by 08-17) | Test stand | Merlin 1C injector-to-manifold joint | Development waits on “an improved method of attaching the injector to the manifold”Open the record | not stated | The record | ||
| 2007 (by 12-10) | Test standTest stand, Texas | Merlin 1C gas generator | Gas generator “blow apart during a start sequence”Open the record | not stated | The record | ||
| 2008 | Test standTest stand, McGregor | Merlin 1C ignition | Ignition on time about 25% of the time; fuel after aborts damaged nozzles. A gas generator blew up when igniter fluid failedOpen the timeline | Igniter fluid flow (one case) | Later accounts | ||
| 2008-06 | GroundIn transit, Honolulu | Kestrel niobium nozzle skirt | Hairline crack in a main weld, found on arrivalOpen the timeline | A road shock in the one leg without an escort | Later accounts | ||
| 2008-08-03 | FlightFlight, stage separation | Falcon 1, Flight 3 | Stages “held together”; no orbitOpen the recordCase 4 | Merlin 1C residual thrust after shutdown pushed the first stage into the second | 4 | 57 | The record |
| 2008-08-03 | CountdownPad, Omelek | Falcon 1, Flight 3 | AbortOpen the record | not stated | 0 | The record | |
| 2008-09 | GroundGround, in the C-17 | Falcon 1 Flight 4 first stage | Not in the record. Later accounts tell of the stage damaged on the flight to Kwajalein (25 PLAN ruling 13 names it)Open the record | not stated | Later accounts | ||
| 2008-09-20 | GroundPad, static fire data | Flight 4 second stage LOX supply line | Data led them “to replace a component”Open the record | not stated. “a good chance we would be ok” | 3 | 8 | The record |
| 2008-11-22 | Test standTest stand, McGregor | Falcon 9 test software | In a 2 a.m. dry run, one line of code opened every valve on the rocketOpen the timeline | Software error | Later accounts | ||
| 2009 to 2012 (date not stated) | Test standTest chamber, Hawthorne | Dragon lithium-polymer battery pack | The pack exploded overnight; “The chamber is totally destroyed”Open the timeline | not stated | Later accounts | ||
| 2010 (date not stated) | Test standDrop test, off Morro Bay | Dragon parachutes | Chutes stayed inflated after splashdown; Dragon sailed toward Diablo CanyonOpen the timeline | Wind | Later accounts | ||
| 2010 (early) | GroundGround, qualification | Falcon 9 flight termination batteries | Batteries failed Air Force temperature-cycle testsOpen the timeline | not stated | Later accounts | ||
| 2010 (five days before a Falcon 9 static fire) | GroundGround, McGregor | Falcon 9 engine controller | “they poured, a quart of water out of the engine controller”Open the timeline | Water ingress in the box design, shared by most avionics boxes | Later accounts | ||
| 2010 to 2012 (date not stated) | Test standGround, EMI test | Dragon sensors | Sensors “very susceptible to EMI”; “it would’ve been a mission failure”Open the timeline | Electromagnetic interference | Later accounts | ||
| 2010-02 (late) | GroundPad, wet dress rehearsal | Falcon 9 Flight 1 first stage cork | “the cork’s bonding adhesive failed and several panels separated”Open the record | Adhesive failed in the cold of LOX | 70 | The record | |
| 2010-03-09 | CountdownPad, SLC-40 | Falcon 9 Flight 1 static fire | Abort at T-2 sOpen the record | The autostart never commanded the ground isolation valve, so no engine spun up | 2 | 4 | The record |
| 2010-06-04 | Flight | Falcon 9, Flight 1: liftoff roll and upper stage roll | The rocket “rolled a bit more than SpaceX engineers had expected”Open the record | Outside: liftoff roll from engine torque; upper stage roll traced to the roll actuator, which “may have overheated due to radiative heating from the nozzle” (Musk) | 12 | 187 | The record |
| 2010-06-04 | FlightOcean | Falcon 9 Flight 1 first stage recovery | “The stage hit the water hard -- parachute deployment was not observed”Open the record | not stated | 187 | The record | |
| 2010-06-04 | FlightFlight, orbit | Falcon 9 Flight 1 upper stage restart (an extra test) | “it did light the engine. We’re not happy with how it did that.” (Musk)Open the timeline | not stated (“no comment”) | Press of the time | ||
| 2010-06-04 | CountdownPad, SLC-40 | Falcon 9, Flight 1 | AbortsOpen the record | “technical anomalies”: not stated | 0 | The record | |
| 2010-06-04 | Flight | Falcon 9 Flight 1 | A 60° roll in the first two seconds; the upper stage lost roll control before reentry and spun over AustraliaOpen the timeline | Engines slightly misaligned (the liftoff roll) | Later accounts | ||
| 2010-12-06 | GroundPad, photo review | Falcon 9 Flight 2 upper stage nozzle extension | “two small cracks in the aft end”Open the recordCase 5 | Outside: a GN2 vent line made the thin end flutter | 2 | 2 | The record |
| 2010-12-08 | FlightOcean | Falcon 9 Flight 2 first stage recovery | Not in the record. Outside: ships waited off Cape Hatteras; no outcome givenOpen the timeline | not stated | Press of the time | ||
| 2010-12-08 | CountdownPad, SLC-40 | Falcon 9, COTS Demo 1 | Abort at T-2:48Open the source | “a false abort” from the ordnance interrupter ground feedback | 0 | 0 | The record |
| 2012 to 2013 | Test standTest stand, McGregor | Merlin 1D development | “we blew up a lot of hardware and we tried probably a hundred different combinations”Open the timeline | not stated | Later accounts | ||
| 2012-05-19 | CountdownPad, SLC-40 | Falcon 9, COTS 2 | Abort at ignitionOpen the recordThe docking room | High chamber pressure on engine 5; a faulty valve | 0 | 3 | The record |
| 2012-05-25 | FlightFlight, near the ISS | Dragon, COTS 2 final approach | Not in the record. Later accounts: the Dragon LIDAR saw stray reflections and the approach paused. The docking room (25.6) covers itOpen the recordThe docking room | not stated | 0 | Later accounts | |
| 2012-10-08 | FlightFlight, T+79 s | Falcon 9 CRS-1, engine 1 | Engine 1 “lost pressure suddenly”; shut down; engine bay panels ejectedOpen the recordCase 6 | Record: not stated (on 2012-10-28 still working “to determine root cause”). Outside: “a material flaw that went undetected in the jacket of the Merlin engine” (Shotwell, 2013-02-28) | 144 | 145 | The record |
| 2012-10-08 | FlightFlight, orbit | Orbcomm OG2 prototype, secondary payload | The satellite “didn’t reach proper orbit”Open the recordCase 6 | Record: “the glitch”. Outside: NASA wanted over 99% odds for a restart; LOX gave ~95%, so no second burn | 2 | The record | |
| 2013 | Test standTest stand, McGregor | Falcon 9 v1.1 first stage (CASSIOPE) | “in the test stand for like six months, right, with all kinds of leaks”; engines swapped on the standOpen the timeline | not stated | Later accounts | ||
| 2013-03-01 | FlightFlight, orbit | Dragon, CRS-2 propulsion | “a minor issue with some of Dragon’s oxidizer tanks”; “a thruster issue”. Outside: three of four thruster pods would not initializeOpen the recordBetween cases | Record: “the Dragon prevalves” (2013-06-07). Outside: stuck oxidizer check valves; “The supplier had made mistakes that we didn’t catch” (Musk, 2013-04-04) | 34 | 2 | The record |
| 2013-08 | GroundPad, Vandenberg | Falcon 9 v1.1 second stage LOX fill-and-drain valve | Valve stuck shut; three days to boil off the LOXOpen the timeline | Water in the ground actuation line | Later accounts | ||
| 2013-09-07 | GroundPad, Vandenberg | Vandenberg water deluge system | Water main burst; the site floodedOpen the timelineCase 7 | A trainee closed six valves at once (water hammer) | Later accounts | ||
| 2013-09-29 | FlightFlight, orbit | Falcon 9 v1.1 CASSIOPE upper stage restart | The planned restart did not happenOpen the sourceCase 7 | Igniter fluid lines froze after long exposure to cold oxygen | 53 | 65 | The record |
| 2013-09-29 | FlightFlight, re-entry | Falcon 9 v1.1 CASSIOPE first stage recovery test | Relit twice; second burn “cut short because the rocket was put in a spin”Open the sourceCase 7 | Outside: “spun up due to aero torque, so fuel centrifuged and we flamed out” (Musk) | 0 | The record | |
| 2013-11-25 | CountdownPad, SLC-40 | Falcon 9 v1.1, SES-8 | Abort at T-6:11Open the source | Outside: a telemetry limit on a power supply | 0 | 0 | The record |
| 2013-11-25 | CountdownPad, SLC-40 | Falcon 9 v1.1, SES-8 | Hold, then scrubOpen the source | Outside: an “off-nominal condition” in first stage LOX pressurization | 0 | 3 | The record |
| 2013-11-26 | Test standTest site, McGregor | Nitrogen tanks for slosh-baffle tests | Two tanks “over pressurized and ruptured” (sheriff); no one hurtOpen the timeline | Outside: “likely the result of over-pressurized gas”; root cause not published | Press of the time | ||
| 2013-11-28 | CountdownPad, SLC-40 | Falcon 9 v1.1, SES-8 | Abort with engines running: slow thrust rampOpen the source | Outside: “oxygen in ground side TEA-TEB” | 2 | 5 | The record |
| 2013-11-28 | CountdownPad, SLC-40 | Falcon 9 v1.1, SES-8 | Scrub at T-48 sOpen the source | The review of A16 was not done | 0 | 5 | The record |
Where failures happened, by year
Test stand and ground fill 2003 to 2005; flight fills 2006 to 2008; from 2010 the flight lane holds failures the rocket survived.
Days from failure to cause, and to the next try
To the cause To the next try
The gap to the next try did not shrink every time. It shrank when it had to.
“If it’s an all-or-nothing proposition then we’ve failed. But it’s not all or nothing.”