At element61, we followed the Databricks Tech Kickoff FY26 to stay up to date with the latest announcements and best practices in data and AI. From Mosaic AI, SAP Databricks to AI-powered BI dashboards, the event covered important technical developments shaping the industry. In this insight, we share key trends and best practices to help businesses make the most of these innovations.
Databricks' AI/BI Dashboards and Genie are disrupting Power BI
Databricks launched in 2023 its AI/BI Dashboards and announced Genie, an AI assistant within Databricks to query your data & your insights. With Genia, you can literally, using English or natural language, run an end-to-end analysis from scratch without coding - IMPRESSIVE!
A common question we often receive is whether Databricks Dashboards will compete with other visualization tools like Power BI or Tableau. Will it not overlap too much & when do we use what? Our reflection: As element61, we don’t see organizations fully pivot from a Power BI or Tableau to Databricks' Dashboard. Rather, for almost all organizations, Databricks’ Genie & the AI/BI dashboard have a role to play alongside the already existing enterprise reporting tools like Power BI. Databricks' AI/BI Dashboards & Genie certainly add value for data engineers, data analysts & champion users working daily hands-on with data.
What's clear is that Databricks’ continuous improvement to its dashboarding features, including the decision to decommission legacy dashboards, highlights their commitment to making this aspect of the Data Intelligence Platform a leading choice for visualizations and data democratization.
In the Tech Summit of Databricks, the discussion around AI/BI Dashboards and Genie highlighted where the Dashboards & Genie application can add the most value & what features are coming.
For Genie to be reliable, structured Genie spaces are essential. After all, querying your data is key, but inconsistent results from Genie can undermine user trust. How do we balance value & risk? Genie spaces can be a valuable asset, but the GIGO (Garbage In, Garbage Out) principle applies: if the data is not properly structured in your Unity Catalog, the value and accuracy of Genie’s outputs will be compromised.
To get the most out of AI/BI Dashboards and Genie, keep the following key practices in mind:
- Genie Spaces require a framework with best practices, like example SQL for FAQS and context-based query enhancements
- Limited UI access is possible & can restrict users to Genie Spaces only without exposing the full Databricks UI. This lowers the threshold to accessing this valuable tool
- For datasets under 100k rows, caching eliminates unnecessary compute. Don't make your datasets too big and when you keep this you'll help manage memory and processing power, thus helping limit costs.
Image
SAP Databricks: Is it the best of both worlds
At element61, one of the key discussions revolves around how to handle SAP data within a modern data platform. This is where the SAP Databricks announcement comes into play. SAP Databricks is a collaborative data platform (part of SAP Business Data Cloud) that combines the powerful data processing and machine learning capabilities of Databricks with SAP's enterprise resource planning (ERP) and business data. This integration enables businesses to seamlessly manage, analyze, and extract insights from the data stored in SAP systems, where much of the critical data businesses generate is still housed.
By leveraging Databricks' advanced analytics and AI capabilities, companies can optimize their data workflows, enhance decision-making, and improve overall efficiency. Connecting SAP's business data with Databricks' scalability and flexibility offers businesses a significant advantage. For more details, feel free to check the insight 'What Databricks-fans should know about SAP Business Data Cloud' from our colleagues at element61.
In the Tech Summit, a series of demos were given that showed the integration of Databricks in the broader platform & how data integration was simplified using Delta Sharing. A lot of interesting stuff... keen to know more: reach out via info@element61.be!
Databricks Apps truly democratise app development
At element61, this is a topic we have been investigating extensively. Databricks Apps provide an interactive platform that empowers non-technical users to access data and custom tools. These applications are designed to leverage datasets for analytics and support critical business decision-making with the power of AI. A major focus is lowering the barrier to creating data and AI applications, enabling users to develop, deploy, and manage apps in a secure, scalable, and cloud-agnostic environment. Databricks Apps will become generally available (GA) in April 2025.
Databricks has a solid business case: not all data products are (luckily) reports. With Databricks Apps, you can:
- Build a small application with some buttons, entry fields, and a simulation behind
- Build a Gen AI app on top of your data
- Build an AI directly on top of your data (without writing technical code)
Exciting product that we see taking a solid part of any Modern data Platform architecture!
On Databricks Apps, the tech Summit focused on the recent announcements, primarily addressing the limitations identified during last year's preview phases.
Limitations of Databricks Apps
- Only supports Python-based frameworks (e.g., Dash, Streamlit)
- Limited to 50 apps per workspace
- Instance type is restricted to 2 vCPUs and 6GB of memory
- No support for API-based authentication
- No support for propagating user permissions to the underlying data objects
Which means (for now)
- Developers familiar with JavaScript will build outside of Databricks Apps
- Limits enterprise-wide adoption of apps
- Cannot support memory-intensive or compute-heavy applications.
- Complex multi-service architectures will be built outside of Databricks Apps
- Apps have a single permission model, where everyone accessing an app can access all objects the app can access, limiting RBAC possibilities
And requires as Future Resolutions that Databricks is working on
- JavaScript support is coming, aligning with the core user base
- App limit per workspace will be removed with General Availability (GA)
- More instance types and horizontal scaling will be introduced
- Token-based authentication is coming, addressing API security needs
- “on-behalf-of-user” execution will be implemented in the GA version
Mosaic AI: a Gateway to monitor Gen AI applications
Of course, we need a Gen AI topic as 1 of the key innovations upcoming. At the Databricks Tech Kickoff, Mosaic AI and its AI Gateway (currently in Public Preview) were highlighted as essential tools for ML model governance and performance optimization. Two key features stand out for the AI Gateway:
- Observability of Model Usage: The AI Gateway tracks model usage and logs metrics into reference tables in Unity Catalog. These insights enable governance teams to analyze performance, enforce access policies, and optimize resource allocation
- A/B Testing: Unlike MLflow’s offline evaluations, the AI Gateway supports real-time A/B testing, comparing models under live traffic. It dynamically splits user queries to assess latency, cost, and engagement, ensuring optimal model selection in production. While MLflow focuses more on performance metrics like accuracy and precision, the AI Gateway brings significant value to testing for Generative AI applications
As we notice as element61 that observability of Gen AI isn't easy, we are very excited on this functionality & product. By integrating observability and A/B testing, the AI Gateway enhances decision-making, cost control, and model performance in AI-driven applications.
More to come soon on this!
How to stay on top of this all
The Databricks Tech Kickoff FY26 covered a wealth of best practices, innovations, and strategies - far more than we could capture in a single insight. As businesses look to optimize their data and AI initiatives, staying informed and leveraging the right expertise is key. At element61, we help organizations navigate these advancements, applying industry best practices to drive efficiency and value. Follow our LinkedIn & this website to stay on top of all the latest announcements.
If you’d like to explore how these insights can benefit your business, get in touch with us.
