https://bricksnotes.com/blog Practical writing on Databricks, Spark, Delta Lake, and the craft of data engineering. en Thu, 08 Oct 2026 01:34:24 GMT https://bricksnotes.com/blog/future-of-data-to-decisions-is-autonomous https://bricksnotes.com/blog/future-of-data-to-decisions-is-autonomous Thu, 08 Oct 2026 01:34:24 GMT BricksNotes Editorial Dashboards wait for people. Autonomous systems act on trusted data within clear limits. Here is what that future looks like, five use cases, and a Free Edition lab to build your first decision loop. AI Agents Data Engineering Databricks Future of Data Context Engineering https://bricksnotes.com/blog/debug-slow-spark-stage-databricks https://bricksnotes.com/blog/debug-slow-spark-stage-databricks Wed, 07 Oct 2026 01:37:02 GMT BricksNotes Editorial When a Spark job suddenly slows down, reading the code is rarely enough. Learn to find the slowest stage, read the plan, spot data skew in the task chart, and choose the right fix. Spark Performance Databricks Debugging Data Engineering https://bricksnotes.com/blog/data-engineering-50-problems-solutions https://bricksnotes.com/blog/data-engineering-50-problems-solutions Tue, 06 Oct 2026 02:07:42 GMT BricksNotes Editorial Duplicates, skew, small files, schema changes, late data, slow joins, high costs. Most production issues repeat. This field guide pairs 50 real problems with the solution, the reason it works, and the chapter where you can practice it. Data Engineering Databricks Performance Data Quality Best Practices https://bricksnotes.com/blog/databricks-ai-decide-explained https://bricksnotes.com/blog/databricks-ai-decide-explained Sun, 04 Oct 2026 15:19:13 GMT BricksNotes Editorial ai_decide is a new Databricks AI Function that answers choice questions with probabilities, not text. Learn its three question types, real SQL patterns and how to route by confidence. AI and data Databricks SQL context-advantage https://bricksnotes.com/blog/context-engineering-model-tools-guardrails https://bricksnotes.com/blog/context-engineering-model-tools-guardrails Sun, 04 Oct 2026 01:46:27 GMT BricksNotes Editorial A model alone is not an agent. The choices around it, context, tools, instructions and guardrails, decide how useful it is and how much control the user keeps. AI and data context-advantage Agents https://bricksnotes.com/blog/databricks-genie-explained-ai-analyst https://bricksnotes.com/blog/databricks-genie-explained-ai-analyst Fri, 02 Oct 2026 03:43:32 GMT BricksNotes Editorial Databricks Genie turns business questions into SQL on your warehouse. The SQL is rarely the hard part. The hard part is teaching Genie what your business terms mean, the same way you would onboard a new analyst. AI and data context-advantage Genie https://bricksnotes.com/blog/when-two-ai-agents-disagree https://bricksnotes.com/blog/when-two-ai-agents-disagree Thu, 01 Oct 2026 12:30:00 GMT BricksNotes Editorial One agent says approve. The other says reject. Both followed their instructions. What decides the outcome is not intelligence, it is shared context, clear authority, and a safe way to escalate to a human. A practical look at multi-agent system design, with a refund example and a Databricks Free Edition exercise. AI and data context-advantage https://bricksnotes.com/blog/superintelligence-foundation-is-data https://bricksnotes.com/blog/superintelligence-foundation-is-data Wed, 30 Sep 2026 13:47:49 GMT BricksNotes Editorial Experts have stopped saying AI and started saying superintelligence. The name changed. The dependency did not. Whoever owns trusted, well defined, governed data still decides how useful intelligent systems become. A long read for data engineers, analysts and everyone who keeps the numbers honest. AI and data context-advantage https://bricksnotes.com/blog/databricks-row-zero-genie-spreadsheets https://bricksnotes.com/blog/databricks-row-zero-genie-spreadsheets Wed, 30 Sep 2026 06:30:00 GMT BricksNotes Editorial Databricks acquired Row Zero to end the spreadsheet export. Here is what live governed spreadsheets change for finance, analysts and AI agents. Genie Governance Unity Catalog The Context Advantage Spreadsheets Data + AI updates https://bricksnotes.com/blog/ai-answers-fast-data-engineering-trust https://bricksnotes.com/blog/ai-answers-fast-data-engineering-trust Tue, 29 Sep 2026 02:00:00 GMT BricksNotes Editorial An AI assistant can answer in one second. Whether that answer deserves trust is decided long before the question is asked, by six basics of data engineering. Data + AI Updates Data Quality Governance Context Advantage Career https://bricksnotes.com/blog/governed-ai-spreadsheets-trusted-metrics https://bricksnotes.com/blog/governed-ai-spreadsheets-trusted-metrics Mon, 28 Sep 2026 02:00:00 GMT BricksNotes Editorial Three spreadsheets, three revenue numbers, zero mistakes. Why AI makes shared definitions matter more, and how to practice it in Free Edition. Genie Governance Semantic Layer Data + AI Updates Unity Catalog Context Advantage https://bricksnotes.com/blog/when-ai-agents-reach-trillion-token-scale https://bricksnotes.com/blog/when-ai-agents-reach-trillion-token-scale Thu, 24 Sep 2026 01:10:00 GMT Team BricksNotes What a trillion-token clinical AI system teaches data teams about trusted context, durable state, controlled access, observability, and cost. AI agents Unity Gateway Lakebase Observability The Context Advantage Data engineering https://bricksnotes.com/blog/genie-one-mcp-enterprise-context https://bricksnotes.com/blog/genie-one-mcp-enterprise-context Wed, 23 Sep 2026 01:04:00 GMT Team BricksNotes Learn how Genie One MCP connects different AI assistants to shared business meaning, richer analytical results, and governed access through Unity Gateway. Genie One MCP enterprise context AI agents Unity Gateway Genie Ontology https://bricksnotes.com/blog/enterprise-context-ai-agents https://bricksnotes.com/blog/enterprise-context-ai-agents Tue, 22 Sep 2026 01:27:00 GMT Team BricksNotes Ali Ghodsi says enterprise context is the missing piece between chatbots and autonomous agents. Learn the architecture, controls, and practical steps behind it. enterprise AI context engineering AI agents Genie Ontology semantic understanding https://bricksnotes.com/blog/gray-failures-anomaly-detection-databricks https://bricksnotes.com/blog/gray-failures-anomaly-detection-databricks Sun, 20 Sep 2026 17:00:00 GMT Team BricksNotes Gray failures slip past healthy dashboards and quietly cost you customers. Learn the four-stage RADAR pattern Databricks uses, and try the core idea in Free Edition. observability anomaly-detection reliability data-engineering https://bricksnotes.com/blog/match-recognize-sql-pattern-matching-explained https://bricksnotes.com/blog/match-recognize-sql-pattern-matching-explained Sat, 19 Sep 2026 13:00:00 GMT Team BricksNotes MATCH_RECOGNIZE is a new Databricks SQL operator that finds event sequences, like a regular expression for rows. Learn the pattern syntax with runnable examples. SQL Databricks Spark SQL Analytics https://bricksnotes.com/blog/enterprise-ai-semantic-understanding-data-model https://bricksnotes.com/blog/enterprise-ai-semantic-understanding-data-model Fri, 18 Sep 2026 01:20:00 GMT Team BricksNotes Enterprise AI does not need another pile of data. It needs shared business definitions that agents can inspect, explain, and use safely. Enterprise AI Semantic Layer Data Modeling Databricks AI Agents Governance https://bricksnotes.com/blog/spark-structured-streaming-state-repartitioning https://bricksnotes.com/blog/spark-structured-streaming-state-repartitioning Thu, 17 Sep 2026 02:20:32 GMT Team BricksNotes A practical guide to on-demand state repartitioning in Apache Spark Structured Streaming on Databricks, including the checkpoint trap, RocksDB requirements, code, monitoring, and a safe rollout plan. Databricks Apache Spark Structured Streaming RocksDB Performance https://bricksnotes.com/blog/shared-context-to-trusted-action-genie-ontology https://bricksnotes.com/blog/shared-context-to-trusted-action-genie-ontology Wed, 16 Sep 2026 02:32:00 GMT Team BricksNotes Why Genie One and Genie Ontology matter beyond question answering, and how shared context can shorten the path from a business signal to a trusted action. Genie AI Agents Context Engineering Governance Databricks https://bricksnotes.com/blog/databricks-genie-one-explained https://bricksnotes.com/blog/databricks-genie-one-explained Mon, 14 Sep 2026 14:35:00 GMT Team BricksNotes Genie One is not a smarter search box. It is an AI coworker that reads your governed data, drafts the SQL, and can take the next step. Here is how it works, and what still belongs to you. genie ai databricks governance sql https://bricksnotes.com/blog/ai-agent-memory-vs-business-state https://bricksnotes.com/blog/ai-agent-memory-vs-business-state Sun, 13 Sep 2026 22:55:00 GMT Team BricksNotes An AI agent can remember a promise without making it true. Here is how systems of record, governed state changes, and audit trails keep agent actions reliable. AI Agents Data Engineering Architecture Governance Databricks https://bricksnotes.com/blog/liquid-clustering-vs-partitioning-explained https://bricksnotes.com/blog/liquid-clustering-vs-partitioning-explained Sun, 13 Sep 2026 04:33:23 GMT BricksNotes Editorial Partitioning feels like the obvious answer until the folders multiply and queries slow down. Here is what is really happening, and what liquid clustering does differently. databricks delta-lake performance liquid-clustering data-engineering https://bricksnotes.com/blog/managing-ai-agent-fleet-choice-context-control https://bricksnotes.com/blog/managing-ai-agent-fleet-choice-context-control Sat, 12 Sep 2026 03:57:37 GMT BricksNotes Enterprises are adopting multiple AI coding tools at once. Here is a calm, practical way to think about managing that fleet: choice of models, shared governed context, and one control plane. ai-agents governance context-advantage databricks enterprise-ai https://bricksnotes.com/blog/slowly-changing-dimensions-explained https://bricksnotes.com/blog/slowly-changing-dimensions-explained Fri, 11 Sep 2026 01:37:54 GMT BricksNotes Editorial A customer moved from Pune to Mumbai. Simple in life, quietly difficult in data. What your table chooses to forget decides what your reports can prove. Data Modeling Delta Lake Fundamentals https://bricksnotes.com/blog/incremental-processing-explained https://bricksnotes.com/blog/incremental-processing-explained Thu, 10 Sep 2026 02:21:09 GMT BricksNotes Editorial You have a billion rows and a million new ones arrive daily. Would you reprocess everything? This explainer covers change detection, merge (upsert) logic, late-arriving data, overlap windows, idempotency, checkpoints, and how it maps to Bronze, Silver, and Gold. data-engineering databricks delta-lake fundamentals https://bricksnotes.com/blog/30000-practice-exams-bricksnotes-milestone https://bricksnotes.com/blog/30000-practice-exams-bricksnotes-milestone Wed, 09 Sep 2026 03:02:55 GMT BricksNotes Editorial More than 30,000 practice exams have been completed on BricksNotes in the last 9 months. Thank you to every learner preparing quietly, one concept at a time. Community Milestone https://bricksnotes.com/blog/data-ai-space-exploration https://bricksnotes.com/blog/data-ai-space-exploration Wed, 09 Sep 2026 01:28:01 GMT BricksNotes Space missions are data missions. Here is how telemetry, science payloads, versioned history and onboard models fit together, and how a lakehouse supports all of it. data engineering AI space vision https://bricksnotes.com/blog/small-files-problem-explained https://bricksnotes.com/blog/small-files-problem-explained Tue, 08 Sep 2026 02:53:30 GMT BricksNotes A pipeline can process the correct data and still be painfully slow. The cause is often hundreds of thousands of tiny files. Learn why the small files problem happens, how to spot it, and how to fix it with partitioning, OPTIMIZE, and better write habits. spark databricks delta lake performance https://bricksnotes.com/blog/personal-ai-agents-2026-data-future https://bricksnotes.com/blog/personal-ai-agents-2026-data-future Mon, 07 Sep 2026 03:36:36 GMT BricksNotes Personal AI agents are coming fast, and they all run on trustworthy data. What the agent era means for data engineering careers, and how to start building toward it today. AI data engineering agents career vision https://bricksnotes.com/blog/medallion-architecture-explained https://bricksnotes.com/blog/medallion-architecture-explained Sun, 06 Sep 2026 02:59:34 GMT BricksNotes Medallion architecture explained in plain words: what bronze, silver, and gold tables are really for, a worked example, the mistakes beginners make, and when fewer layers are the better design. databricks delta lake medallion architecture fundamentals https://bricksnotes.com/blog/data-agents-future-of-data-engineering https://bricksnotes.com/blog/data-agents-future-of-data-engineering Sat, 05 Sep 2026 05:26:55 GMT BricksNotes AI agents are starting to work with data directly. This article looks at what data agents are, why trustworthy tables matter more than ever, and how a beginner today can prepare for the agent era. AI Data Engineering Vision Agents https://bricksnotes.com/blog/data-ai-golden-era-vision https://bricksnotes.com/blog/data-ai-golden-era-vision Fri, 04 Sep 2026 17:10:12 GMT BricksNotes The future belongs to data and AI, and the bottleneck is no longer the model. It is trustworthy data. A look at why this era rewards builders, the honest skill path from foundations to AI readiness, and the BricksNotes vision behind it. data engineering AI vision career https://bricksnotes.com/blog/data-engineering-faster-decisions https://bricksnotes.com/blog/data-engineering-faster-decisions Thu, 03 Sep 2026 02:11:09 GMT BricksNotes Your pipeline runs in four minutes and the decision still waits for Monday. A practical look at decision latency, designing backwards from the decision, and what changes when the consumer is an agent. data engineering decisions databricks https://bricksnotes.com/blog/databricks-small-file-problem-optimize-vacuum https://bricksnotes.com/blog/databricks-small-file-problem-optimize-vacuum Wed, 02 Sep 2026 03:10:00 GMT BricksNotes A Delta table that was fast in week one became slow by month three. The data had not grown much. The file count had. Here is what the small file problem is, why streaming and frequent merges cause it, and how OPTIMIZE and VACUUM keep a table healthy. databricks delta lake performance fundamentals https://bricksnotes.com/blog/delta-lake-time-travel-explained https://bricksnotes.com/blog/delta-lake-time-travel-explained Tue, 01 Sep 2026 03:38:53 GMT BricksNotes Delta Lake keeps every version of your table. Learn how to query older versions with VERSION AS OF, read table history, and undo bad writes with RESTORE, with simple examples that run in Databricks Free Edition. Databricks Delta Lake time travel data engineering https://bricksnotes.com/blog/databricks-pipeline-observability-alerts-runbook https://bricksnotes.com/blog/databricks-pipeline-observability-alerts-runbook Mon, 31 Aug 2026 01:51:24 GMT BricksNotes A green run is a weak signal. Here are the four questions your pipeline should answer on its own, with freshness checks, run metrics, Delta constraints and alerts you can build in Free Edition. Databricks monitoring data-quality pipelines https://bricksnotes.com/blog/databricks-electric-acquisition-lakebase-agents https://bricksnotes.com/blog/databricks-electric-acquisition-lakebase-agents Sun, 30 Aug 2026 02:25:52 GMT BricksNotes Databricks acquiring Electric is a bet that agents need data close to where they run. Here is what it means for Lakebase, and what it changes for data engineers. Databricks ai-agents lakebase architecture https://bricksnotes.com/blog/what-is-lakebase-explained-simply https://bricksnotes.com/blog/what-is-lakebase-explained-simply Sat, 29 Aug 2026 02:56:00 GMT BricksNotes Lakebase is managed Postgres inside Databricks. Here is the idea behind it, where it fits, the use cases worth trying, and why agents make it more than a convenience feature. Databricks Lakebase Data + AI Architecture Agents https://bricksnotes.com/blog/databricks-query-optimization-execution-plan https://bricksnotes.com/blog/databricks-query-optimization-execution-plan Fri, 28 Aug 2026 01:42:09 GMT BricksNotes Two queries, identical results, wildly different cost. Here is what happens between the SQL you write and the answer you get, and the seven techniques that decide how much work the engine does. Databricks Spark Performance SQL Data Engineering https://bricksnotes.com/blog/databricks-broadcast-vs-shuffle-joins https://bricksnotes.com/blog/databricks-broadcast-vs-shuffle-joins Thu, 27 Aug 2026 02:09:50 GMT BricksNotes One line joined orders to customers, and an eleven minute job ran all night. Here is what a join really does on a cluster, and how to choose the right one. Databricks Spark Performance Data Engineering https://bricksnotes.com/blog/schema-changes-data-contracts-databricks https://bricksnotes.com/blog/schema-changes-data-contracts-databricks Wed, 26 Aug 2026 01:34:12 GMT BricksNotes Schema evolution is normal. Uncontrolled schema evolution is dangerous. The five change types, why a data contract does what a schema cannot, and six habits that make change visible, testable and safe in Databricks. Databricks Schema Evolution Data Quality Data Engineering https://bricksnotes.com/blog/databricks-duplicate-records-deduplication https://bricksnotes.com/blog/databricks-duplicate-records-deduplication Tue, 25 Aug 2026 00:34:08 GMT BricksNotes A pipeline can finish green and your numbers can still be wrong. Duplicates rarely break pipelines. They break business decisions. Here is how to handle them properly. Databricks Data Quality SQL Fundamentals https://bricksnotes.com/blog/databases-built-for-agents-not-just-developers https://bricksnotes.com/blog/databases-built-for-agents-not-just-developers Mon, 24 Aug 2026 00:53:14 GMT BricksNotes For forty years databases were designed around a human who could ask a teammate. Agents cannot. Here is how a data platform changes when machines are the main readers, and what to do about it in Databricks this week. AI Agents Unity Catalog Data Engineering Databricks https://bricksnotes.com/blog/databricks-data-skew-slow-tasks https://bricksnotes.com/blog/databricks-data-skew-slow-tasks Sun, 23 Aug 2026 03:34:15 GMT BricksNotes The code did not change. The data did. Here is how one heavy key turns a nine minute job into a fifty one minute one, and how to fix it. Databricks Spark Performance Joins https://bricksnotes.com/blog/csv-json-parquet-delta-which-format-to-use https://bricksnotes.com/blog/csv-json-parquet-delta-which-format-to-use Sat, 22 Aug 2026 15:16:16 GMT BricksNotes Editorial CSV, JSON, Parquet, or Delta? A practical comparison with runnable Databricks Free Edition examples, real size and speed differences, and a simple rule for each layer of your pipeline. Delta Lake Parquet File Formats Data Engineering https://bricksnotes.com/blog/bricksnotes-40000-to-100000-learners https://bricksnotes.com/blog/bricksnotes-40000-to-100000-learners Fri, 21 Aug 2026 23:57:32 GMT BricksNotes Editorial A hundred thousand data professionals have now learned with BricksNotes. Here is the honest story of how it happened, what we built along the way, and why we are not stopping. Community Milestone Data Engineering 2026 https://bricksnotes.com/blog/databricks-data-quality-checks https://bricksnotes.com/blog/databricks-data-quality-checks Fri, 21 Aug 2026 02:21:49 GMT BricksNotes Editorial A successful run tells you the code finished, not that the data is right. How to write quality rules, decide what happens to bad rows, and enforce them with Delta constraints and pipeline expectations. Databricks Data Quality Delta Lake Pipelines https://bricksnotes.com/blog/built-first-data-engineering-project-two-evenings https://bricksnotes.com/blog/built-first-data-engineering-project-two-evenings Thu, 20 Aug 2026 13:30:43 GMT BricksNotes Editorial A boundary instead of a study plan: two evenings, 100 messy orders, one gold table, and a pipeline that runs twice without breaking. Here is the full walkthrough you can repeat tonight in Databricks Free Edition. Databricks Data Engineering PySpark Getting Started 2026 https://bricksnotes.com/blog/databricks-late-arriving-data https://bricksnotes.com/blog/databricks-late-arriving-data Thu, 20 Aug 2026 01:50:50 GMT BricksNotes Editorial A phone was offline, a partner file was late, a queue backed up. Here is how to keep event time honest and stop numbers from drifting quietly. Databricks Streaming Delta Lake Data Quality https://bricksnotes.com/blog/databricks-backfill-without-breaking-downstream https://bricksnotes.com/blog/databricks-backfill-without-breaking-downstream Wed, 19 Aug 2026 01:51:14 GMT BricksNotes Editorial Reloading March means touching a table six dashboards already read. Here is the plan that makes a backfill boring instead of frightening. Databricks Delta Lake Backfill Pipelines