Data and AI in space exploration: from orbit to answer

Telemetry that never stops, science files that arrive late, and models that must decide alone. Space is where data engineering has nowhere to hide.

Space missions have always been data missions. A spacecraft cannot be visited, repaired by hand, or asked how it feels. Everything we know about it arrives as numbers. Everything we learn from it arrives as numbers too.

That is why space exploration is one of the most interesting places to think about data and AI together. There is no shortcut around good data engineering here. A rover cannot wait three days for a spreadsheet to be cleaned. A telescope cannot store every photon it sees. A satellite cannot phone home for permission before avoiding a piece of debris.

So the question becomes practical. What has to happen close to the spacecraft, what has to happen on the ground, and how do we build a system where both sides trust the same history?

The shape of space data

Start with the raw material. A modern mission produces a few very different kinds of data at the same time.

Telemetry is the heartbeat. Thousands of small readings a second: voltages, temperatures, battery state, wheel current, antenna pointing. Each reading is tiny. Together they are enormous, and they arrive as an endless stream.

Science payload data is the reason the mission exists. Images, spectra, radar returns, particle counts. These files are large and lumpy, and they arrive in bursts whenever a link is available.

Ground and operations data is the context. Commands that were sent, contact windows, weather at the ground station, calibration campaigns, engineer notes.

And then there is the awkward category: the data that arrives late, out of order, or twice. A pass gets replayed. A packet is recovered from a buffer a week later. A file is retransmitted because the first copy was corrupted.

If that last paragraph made you nod, you already understand why space is a good teacher. These are the same problems every serious data platform has, only with the excuses removed.

From orbit to answer

The path from a signal to a decision is longer than most people imagine, and it looks a lot like the medallion pattern.

The bronze layer is the honest record. Every packet, every file, exactly as it was received, with the time it was received and which ground station received it. Nothing is corrected here. If the data was ugly, the ugliness is preserved, because in a mission the ugliness is often the finding.

The silver layer is where the data becomes usable. Packets are decoded into named readings. Timestamps are converted to a single reference clock. Duplicate replays are collapsed. Units are made consistent. Calibration is applied so a raw count becomes a temperature.

The gold layer is where the mission asks questions. Battery health per orbit. Instrument uptime per week. A sky survey table that scientists can join against. A daily view an operations team actually looks at before a pass.

If you want to build that shape with your own hands, the medallion architecture lesson walks through it with small tables you can create in an evening, and the medallion article explains why the layers exist at all.

Why history matters more here than anywhere

Space data has a habit of being reinterpreted. A calibration curve is corrected two years after launch. A sensor is found to drift with temperature. A software patch changes how a value is reported. Suddenly every conclusion built on the old numbers has to be checked.

This is the reason Delta Lake style tables fit missions so well. When each write is a version, you can ask what the table looked like on the day a decision was made, and you can compare it with what the table says now. That is not a nice extra. In science it is the difference between a result and a rumour.

The practical habits are simple. Never overwrite the raw record. Reprocess into a new version of the derived tables. Keep the recipe that produced each version. Then, when someone asks why the answer changed, you can show them instead of guessing.

The Delta Lake lesson covers the mechanics, and time travel explained shows how to look backwards without breaking anything.

Streams that never stop

Telemetry does not have a start and an end. It just keeps coming. That means the pipeline has to be written as something that runs forever and picks up where it left off, not as a script somebody runs each morning.

Two ideas carry most of the weight. First, incremental processing: only handle what is new, and remember what you already handled. Second, tolerance for late data: assume that a reading from Tuesday may show up on Friday and that the pipeline must fold it in without a rebuild.

Both of these are ordinary data engineering skills, and both are practised directly in the incremental processing lesson and the streaming lesson. The article on late arriving data is the closest thing to a mission operations story we have written.

There is also a very physical reason to care about file layout. Telemetry written every few seconds creates millions of tiny files, and tiny files are the quiet killer of query speed. Anyone building a stream should read the small files problem before the problem finds them.

The interesting part: deciding where the intelligence lives

Here is the constraint that makes space different. The link is narrow and the delay is real. Light takes minutes to reach Mars, and much longer to reach the outer planets. You cannot put a human in a loop that needs to close in one second.

So intelligence gets split.

Onboard, the job of a model is to look and choose. Is this image mostly cloud, or is there something worth spending bandwidth on? Does this vibration pattern look like the failure signature we saw in testing? Should the camera take a second look at that plume right now, before the moment passes? The model onboard is small, careful, and rehearsed. Its main power is throwing away the boring parts of the universe so the interesting parts can be sent home.

On the ground, the job is memory and judgement. This is where every version of every reading is kept, where models are trained and retrained, where a human can ask an open question that nobody planned for. The ground side does not need to be fast in milliseconds. It needs to be complete and honest.

The bridge between them is the part people underestimate. A model onboard is only as good as the labelled history that trained it, and that history lives in the lakehouse. Every onboard decision should also be logged and sent home, so the ground can measure whether the model was right. Without that loop you have automation. With it you have learning.

Futuristic, but not fictional

It is worth being specific about what a data and AI stack could unlock in the next decade of exploration.

Autonomous science selection. A spacecraft that ranks its own observations and downlinks the best ones first, instead of sending everything and letting a queue decide.

Predictive health for hardware nobody can touch. Not a threshold alarm, but a model that says this reaction wheel is behaving like the one that failed on a sister mission, three months before failure.

Constellation level thinking. Hundreds of small satellites treated as one instrument, with a shared table of what each of them saw, so a question can be answered across the whole fleet rather than one spacecraft at a time.

Conversational mission archives. Decades of data from many missions, indexed well enough that a researcher can ask a plain question and get a grounded answer with the source rows attached.

Simulation that feeds on reality. Digital twins of a spacecraft trained on its own telemetry, used to rehearse a risky manoeuvre before it is commanded.

None of these are blocked by imagination. They are blocked by data plumbing. Every one of them assumes clean, versioned, well described tables and a team that trusts them.

Where Databricks fits

Nothing here needs a special space platform. The needs map onto features that already exist.

Delta tables give the versioned, correctable history a mission needs. Structured streaming and Lakeflow Declarative Pipelines handle telemetry that never stops. Expectations and quality checks catch a sensor that has started lying, which is the space version of a data quality bug. Unity Catalog gives one place to describe what every table means and who may see it, which matters when a mission has partners across institutions and embargo dates on science data. Volumes hold the raw payload files next to the tables that describe them. Model serving and the AI gateway put a trained model behind an interface that an operations tool can call. Genie style natural language querying turns a mission archive into something a scientist can interrogate without waiting for an engineer.

If you want to feel this instead of reading it, the practical order is small. Start with Start Here to get a workspace and a first table. Learn Delta Lake so history stops being scary. Build the layers in medallion architecture. Add data quality checks so bad readings announce themselves. Govern it in Unity Catalog. Then look at AI gateway and serving and Genie spaces to see how a model and a question sit on top.

The honest takeaway

The romantic view of space is the launch. The working view is a table that can be trusted a decade later.

If you learn to build pipelines that keep raw truth, correct themselves without lying about the past, and hand clean tables to a model, you are practising the same craft that missions depend on. The scale is different. The thinking is identical.

That is really the whole idea behind this book. Understand how data systems behave, and the domain becomes a detail. Space just happens to be the domain where the consequences are impossible to hide.

Continue learning

• Medallion architecture explained simply
• Delta Lake time travel explained
• Handling late arriving data
• The small files problem
• The next phase of data engineering is faster decisions

If this made you want to build something, start with Start Here. Bring your curiosity, we will bring the tables. See you in the lessons.