Skip to content

drcaiomoreno/Generative-AI-Using-Databricks

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
did-bcn-april-2025
did-bcn-april-2025
 
 
.DS_Store
.DS_Store
 
 
Caio Moreno - DataSaturdayMadrid 2024 - Final Presentation - 30.11.2024 - External Version.pdf
Caio Moreno - DataSaturdayMadrid 2024 - Final Presentation - 30.11.2024 - External Version.pdf
 
 
Generative AI - DevScope - Oporto - Portugal - 12 March 2024 - Caio Moreno - Final.pdf
Generative AI - DevScope - Oporto - Portugal - 12 March 2024 - Caio Moreno - Final.pdf
 
 
Generative-ai-DataHour-India-10Jan2024-CaioMoreno.pdf
Generative-ai-DataHour-India-10Jan2024-CaioMoreno.pdf
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
gen-ai-materials.MD
gen-ai-materials.MD
 
 
llm-for-recommendations-systems.MD
llm-for-recommendations-systems.MD
 
 
small-llms.md
small-llms.md
 
 

Repository files navigation

Generative AI / LLMs using Databricks Data Intelligence Platform

This repo aims to help the open source developer Generative AI / LLM Community to use Databricks to build Generative AI Solutions.

https://www.databricks.com/discover/generative-ai

Databricks Intelligence Day - Barcelona - April 2025

https://github.com/drcaiomoreno/Generative-AI-Using-Databricks/tree/main/did-bcn-april-2025/notebook-demo

Databricks Generative AI Cookbook

https://ai-cookbook.io/index.html

Gen AI / Databricks

https://docs.databricks.com/en/generative-ai/generative-ai.html
https://www.databricks.com/discover/generative-ai
https://www.databricks.com/blog/databricks-announces-industrys-first-generative-ai-engineer-learning-pathway-and-certification
https://www.databricks.com/blog/building-custom-genai-llms-and-beyond

Generative AI Architecture Patterns

https://www.databricks.com/product/machine-learning/build-generative-ai

ChunkViz

https://github.com/gkamradt/ChunkViz

DBRX

https://www.databricks.com/blog/announcing-dbrx-new-standard-efficient-open-source-customizable-llms?itm_data=home-under-hero-promo

MegaBlocks

MegaBlocks is a light-weight library for mixture-of-experts (MoE) training.
https://arxiv.org/abs/2211.15841
https://github.com/databricks/megablocks

Slides - Data Saturday Madrid 2024 - 30.11.2024

Deploying a generative AI solution in Production with Azure Databricks
https://github.com/drcaiomoreno/Generative-AI-Using-Databricks/blob/main/Caio%20Moreno%20-%20%20DataSaturdayMadrid%202024%20-%20Final%20Presentation%20-%2030.11.2024%20-%20External%20Version.pdf

Slides - Dev Scope / Databricks Event - 12 March 2024

https://github.com/drcaiomoreno/Generative-AI-Using-Databricks/blob/main/Generative%20AI%20-%20DevScope%20-%20Oporto%20-%20Portugal%20%20-%2012%20March%202024%20-%20Caio%20Moreno%20-%20Final.pdf

Slides - Data Hour - 10 Jan 2024

https://github.com/drcaiomoreno/Generative-AI-Using-Databricks/blob/main/Generative-ai-DataHour-India-10Jan2024-CaioMoreno.pdf

Presentation Video / YouTube

https://www.youtube.com/watch?v=StBvyEdg_SE

LLM Book

Natural Language Processing with Transformers: Building Language Applications With Hugging Face
https://www.amazon.com/Natural-Language-Processing-Transformers-Applications/dp/1098103246

Data Hour Video

https://community.analyticsvidhya.com/c/datahour/generative-ai-llms-using-databricks-data-intelligence-platform

Data Hour Talk

https://datahack.analyticsvidhya.com/contest/datahour-generative-ai-llms-using-databricks-data-intelligence-platform/

Addditional links

https://github.com/databricks-academy/llm-foundation-models

Gen AI Labs

  1. Deploy Your LLM Chatbot With Retrieval Augmented Generation (RAG), DBRX Instruct Foundation Models and Vector Search
    https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/lakehouse-ai-deploy-your-llm-chatbot

  2. Creating Brand-Aligned Images Using Generative AI
    https://www.databricks.com/blog/creating-brand-aligned-images-using-generative-ai
    https://databricks-industry-solutions.github.io/personalized_image_gen/#personalized_image_gen.html
    https://dreambooth.github.io/

  3. Fine-Tuning Large Language Models with Hugging Face and DeepSpeed
    https://www.databricks.com/blog/fine-tuning-large-language-models-hugging-face-and-deepspeed

4.Automating Radiology Workflow with Large Language Models on Databricks
https://www.databricks.com/blog/automating-radiology-workflow-large-language-models-databricks

Databricks Architectures

https://www.databricks.com/resources/architectures

Databricks Industry Solutions

https://github.com/databricks-industry-solutions

Databricks Academy Login

https://www.databricks.com/learn/training/login

Generative AI Fundamentals

https://www.databricks.com/resources/learn/training/generative-ai-fundamentals

Databricks Gen AI Learning Offerings:

https://www.databricks.com/blog/now-available-new-generative-ai-learning-offerings

Caio's Gen AI YouTube Playlist

https://www.youtube.com/watch?v=89CTQQzpe1U&list=PL7M0Nkn3oRYVudYfQiQz5cGpTCbxqkmn1

Open LLM Leaderboard

https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

Microsoft Phi-2

https://huggingface.co/microsoft/phi-2
https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/

REVOLUTIONIZING INSURANCE CUSTOMER SERVICE WITH ADVANCED AI CHATBOT (Santa Lucia Gen AI Use Case)

https://www.databricks.com/dataaisummit/session/revolutionizing-insurance-customer-service-advanced-ai-chatbot
https://www.youtube.com/watch?v=4e_v_-f6-Bk

Santalucía Seguros: Enterprise-level RAG for Enhanced Customer Service and Agent Productivity

https://www.databricks.com/blog/santalucia-seguros-enterprise-level-rag

La Voz del CDO con Néstor Álvaro, Head Analítica Avanzada de Santa Lucia Seguros

https://www.youtube.com/watch?v=gxl-mUI3KVA
https://www.youtube.com/watch?v=dT3MmI-Rhek

Six steps to improve your RAG application’s data foundation

https://community.databricks.com/t5/technical-blog/six-steps-to-improve-your-rag-application-s-data-foundation/ba-p/97700

How to integrate Mistral AI multimodal models from Huggingface with Databricks

https://community.databricks.com/t5/technical-blog/how-to-integrate-mistral-ai-multimodal-models-from-huggingface/ba-p/119126

AI/BI Genie + Databricks SQL Serverless + Unity Catalog Demo/Workshop

https://caiomsouza.medium.com/ai-bi-genie-databricks-sql-serverless-unity-catalog-demo-workshop-bbfeb39ae0be

Databricks Genie AI/BI integrate to Microsoft Teams (Conversational AI)

https://caiomsouza.medium.com/databricks-genie-ai-bi-integrate-to-microsoft-teams-conversational-ai-799db59e7d76

DeepSeek R1 on Databricks

https://caiomsouza.medium.com/deepseek-r1-on-databricks-32d8ab912bb4

Get Ready to be Databricks Certified: Generative AI Engineer Associate

https://caiomsouza.medium.com/get-ready-to-be-databricks-certified-generative-ai-engineer-associate-229d6ca1174b

The Shift from Models to Compound AI Systems

https://caiomsouza.medium.com/the-shift-from-models-to-compound-ai-systems-00fb23544ad7

The AI Race, The Stargate Project, DeepSeek Tsunami, Open Source Models and why should I care?

https://caiomsouza.medium.com/the-ai-race-the-stargate-project-deepseek-tsunami-open-source-models-and-why-should-i-care-41533ec6966d

Introducing the Databricks AI Security Framework (DASF) to Manage AI Security Risks

https://caiomsouza.medium.com/introducing-the-databricks-ai-security-framework-dasf-to-manage-ai-security-risks-c9aa32128870

E-books about Apache Spark and Kafka

Spark: The Definitive Guide: Big Data Processing Made Simple
Graph Algorithms: Practical Examples in Apache Spark and Neo4j
Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale

Part 1: Implementing CI/CD on Databricks Using Databricks Notebooks and Azure DevOps

https://www.databricks.com/blog/2021/09/20/part-1-implementing-ci-cd-on-databricks-using-databricks-notebooks-and-azure-devops.html https://github.com/mshtelma/databricks_ml_demo

Real-Time Mode Technical Deep Dive: How We Built Sub-300 Millisecond Streaming Into Apache Spark™

https://www.databricks.com/dataaisummit/session/real-time-mode-technical-deep-dive-how-we-built-sub-300-millisecond

PDF Document Ingestion Accelerator for GenAI Applications

https://www.databricks.com/dataaisummit/session/pdf-document-ingestion-accelerator-genai-applications

About

Generative AI / LLMs using Databricks Data Intelligence Platform

Resources

Readme

License

MIT license
Activity

Stars

13 stars

Watchers

1 watching

Forks

10 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages