Alberto Ramon Portoles

If your dashboards take minutes to load, you’ve already lost. Slow dashboards cost credibility. Snowflake in Australia gives BI speed, Databricks gives ML power. Which one is your team optimised for?

1

📐 What Do Data Architects Earn in the UK in 2025? With businesses scaling faster and data stacks growing more complex, Data Architects are in high demand—and their salaries reflect it. 💡 In our latest guide, you’ll find: ✔️ Average UK salary benchmarks for Data Architects ✔️ How location, industry, and cloud expertise impact earnings ✔️ What senior-level architects can expect in 2025 If you're hiring for this role—or planning your next career move—this breakdown gives you a clear view of what’s competitive. 👉 Read the full report: https://lnkd.in/e7Q7P3C7 Hiring data talent this year? Let’s talk about how Live Digital can help. #DataArchitect #TechSalaries #DataCareers #HiringInTech #LiveDigital #UKTechJobs

1

A detailed post on how to implement a dimensional data warehouse on Data bricks platform. A very useful article and a must read if you are going to undertake a building dimensional data warehouse. This is a multi part article and very written. https://lnkd.in/gGjvS_Cn #databricks #datawarehouse #dimensionalmodeling

3

The gap in Databricks is tribal knowledge.    Not the platform. The context around it.    We've spent years helping clients wrangle  → Data pipelines  → ML workflows,  → Agentic systems    Same problem every time.    Great platform, too much institutional  knowledge needed to move fast.    That changed when I started working with  the Databricks AI Dev Kit.    It gives AI coding agents actual  Databricks context.    What it gives your agent:    50+ live tools.  19 skill files packed with platform patterns.  An MCP server that lets your agent act,  not just suggest.    Claude Code, Cursor, whatever you use.    Pair that with Genie Code and the loop  from idea to working pipeline  tightens dramatically.    At FrameData AI, this is exactly  what lets us show up to client  conversations with something built.    Not a deck.  Not a POC roadmap.  Something real.    No context loss. Faster value.  That's the job.    If you're building data and AI systems  on Databricks, what's your agent stack? 

13

1 Comment

Real data value occurs when teams can act with confidence, not when they can generate another dashboard. Finworks data management supports intelligent discovery, automated transformation, real-time modelling, transparency, analytics, and secure distribution to downstream systems. Aligned to UK Government and Central Banking security standards. If you're modernising data capability, the goal is simple: reliable data that moves safely into decisions and operations. Read more: https://hubs.li/Q045Hkm_0 #DataManagement #DataGovernance #FinancialServices #ExpertInsights #Finworks

4

Principal Azure Data Engineer role at Capco in London 🚀 Lead enterprise-grade data solutions on Databricks and Azure for Tier 1 financial institutions. You'll architect scalable pipelines with DeltaLake, Spark Structured Streaming, and Unity Catalog while mentoring engineering teams. Tech stack: Python, PySpark, Azure DevOps, Unity Catalog, CI/CD pipelines. Hybrid working with 40+ hours annual training and a business coach from day one 💼 Interested in architecting the future of financial data platforms? https://lnkd.in/euDynR86

1

💥 ADF’s Get Metadata Activity Will Break at 250K+ Files - Here's What To Do Instead If you’ve worked on a data migration project, especially involving on-prem folders with thousands of files, you will hit this wall: Get Metadata Activity fails if your folder has more than 250,000 files. Yes, ADF throws a silent error or just keeps spinning. No clear error message. No graceful fallback. Here’s some way's to handle it. The Problem: Using Get Metadata with Field List = Child Items, on small folders works fine. On a folder with 270K files💥 failure. The Fix: Scalable Metadata Handling Instead of relying on a single Get Metadata pull, Split the logic: 1. List folder contents in chunks ~ Use a PowerShell script or Data Flow to paginate file listing 2. Use Filter activity to select only relevant files ~ For example, only .csv or LastModified > X 3. Use ForEach activity with Batch Count = 20 or 50 ~ This ensured parallelism without memory overload 4. Log file count using Append Variable and push it to a control table ~ For audit and retry logic Bonus Tip: If you already know your folder is huge, Avoid Get Metadata entirely. Instead, trigger a Logic App or Azure Function that streams filenames to a control file in ADLS or a SQL table then loop through that list in ADF. Real lesson: Cloud tools give you superpowers but they don’t scale magically. You have to design for scale, not just success. 250K files is not a corner case in the enterprise. Plan for it, before your pipeline breaks silently at 2 AM. #AzureDataFactory #GetMetadataLimit #DataEngineering #ADFScaling #PipelineDesign #CloudProjects

460

15 Comments

 ETL vs ELT – What’s the Difference? Let’s simplify it with apples: ETL (Extract, Transform, Load) You wash, peel, and cut the apples (transform) before putting them in the basket (load). ➤ Example: You clean and process the data before storing it in the data warehouse. ELT (Extract, Load, Transform) You put raw apples straight into the basket (load), then wash and cut them later (transform) when needed. ➤ Example: You load raw data into a cloud data warehouse (like Snowflake or BigQuery) and transform it there. - ETL is used when transformation needs to happen early. - ELT works well with modern data platforms that handle transformation at scale. hashtag #DataEngineering #AzureDataEngineer #Remote

114

3 Comments

🚦 Data Observability is no longer optional. Over the years, I’ve seen too many data platforms fail quietly — not because of a lack of technology, but because nobody was watching the right things. Observability isn’t just monitoring servers. It’s about catching issues before they break trust: ✔️ Schema drift ✔️ Pipeline health ✔️ Performance & latency ✔️ Lineage, access, security, cost, and compliance In short — when data breaks silently, business decisions break loudly. This is why at DataQ Consulting, we emphasize observability as a foundation for reliable, business-ready data. Curious to hear: What’s the hardest thing to observe in your data stack? #DataObservability #DataStrategy #DataQuality #DataQConsulting

3

🧨 𝐁𝐥𝐨𝐚𝐭𝐢𝐧𝐠: 𝐖𝐡𝐞𝐧 𝐏𝐨𝐬𝐭𝐠𝐫𝐞𝐒𝐐𝐋 𝐚𝐧𝐝 𝐎𝐮𝐫 𝐒𝐭𝐨𝐦𝐚𝐜𝐡 𝐓𝐞𝐥𝐥 𝐭𝐡𝐞 𝐒𝐚𝐦𝐞 𝐒𝐭𝐨𝐫𝐲 As a PostgreSQL DBA, one of the quirks I deal with regularly is database bloat. Recently, I came across a video from Dr Palaniappan Manickam (Dr. Pal), who explained stomach bloating in his trademark humorous style. That’s when it hit me — the two worlds are not so different. One happens inside a database, the other inside us. Both slow things down, consume unnecessary space, and frustrate us at the wrong time! 📌 𝐈𝐧 𝐏𝐨𝐬𝐭𝐠𝐫𝐞𝐒𝐐𝐋, 𝐛𝐥𝐨𝐚𝐭 𝐥𝐨𝐨𝐤𝐬 𝐥𝐢𝐤𝐞 𝐭𝐡𝐢𝐬: Dead tuples pile up --> queries start crawling Autovacuum struggles --> disk usage balloons Fixes: VACUUM, REINDEX, partitioning, tuning 📌 𝐈𝐧 𝐨𝐮𝐫 𝐛𝐨𝐝𝐲, 𝐛𝐥𝐨𝐚𝐭 𝐥𝐨𝐨𝐤𝐬 𝐥𝐢𝐤𝐞 𝐭𝐡𝐢𝐬 (𝐃𝐫. 𝐏𝐚𝐥’𝐬 𝐰𝐢𝐬𝐝𝐨𝐦): Bad gut bacteria pile up --> stomach feels “6 months pregnant” Digestion slows --> gas, fullness, discomfort Fixes: mindful diet, hydration, exercise, probiotics 😅 𝐀𝐧𝐝 𝐡𝐞𝐫𝐞’𝐬 𝐭𝐡𝐞 𝐟𝐮𝐧𝐧𝐲 𝐩𝐚𝐫𝐚𝐥𝐥𝐞𝐥: Dead tuples = undigested beans Autovacuum = your body’s metabolism VACUUM FULL = that crash cleanse (works, but locks you out while running!) Reindex = probiotics for your indexes Partitioning = smaller, frequent meals instead of one giant buffet 🔍 𝐌𝐲 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲 Whether it’s a database or the human gut — bloat is inevitable, but fixable. You can’t stop it from happening, but you can monitor, tune, and act at the right time. As Dr. Pal says: “𝐅𝐨𝐨𝐝 𝐢𝐬 𝐦𝐞𝐝𝐢𝐜𝐢𝐧𝐞—𝐲𝐨𝐮 𝐚𝐫𝐞 𝐰𝐡𝐚𝐭 𝐲𝐨𝐮 𝐞𝐚𝐭.” And as DBAs know: “𝐃𝐚𝐭𝐚 𝐢𝐬 𝐦𝐞𝐝𝐢𝐜𝐢𝐧𝐞—𝐲𝐨𝐮 𝐚𝐫𝐞 𝐡𝐨𝐰 𝐲𝐨𝐮 𝐕𝐀𝐂𝐔𝐔𝐌.” I’ll tag Dr Palaniappan Manickam here because it’s rare to find a gastroenterologist and a DBA talking about the same “bloating” problem—just from two different worlds. And honestly, many of us IT folks (especially the non-moving, WFH warriors) need this reminder to tune not just our databases, but also our diets. Do check out his YouTube video on bloating—you’ll laugh, learn, and maybe even VACUUM FULL your stomach! Here’s Dr. Pal’s YouTube talk on bloating - watch it, laugh a little, and learn a lot: https://lnkd.in/gABDbfCN  #PostgreSQL #DatabaseManagement #GutHealth #HealthHumor #TechHumor #LearningFromEverywhere #Bloating #Tamil

24

2 Comments

❄️#Snowflake + dbt = The ELT Power Duo in #2025 #Musk know ---------------------------------------------------------------------------------- . . . . . As a Senior Data Engineer, I’ve seen firsthand how companies are shifting from heavy ETL to ELT (Extract → Load → Transform) — and the Snowflake + dbt combo makes this shift seamless. Here’s how we recently used it in a real project: 🔹 Pipeline Architecture 1️⃣ Data Ingestion Raw data (JSON, CSV, API feeds) ingested into Snowflake RAW schema via Fivetran / Airbyte. 2️⃣ Transformation with dbt stg_ models → Clean & standardize source data int_ models → Apply joins, enrichments, calculations fct_ & dim_ models → Final fact/dimension tables for analytics 3️⃣ Automation Orchestrated dbt runs with Airflow + GitHub Actions Implemented incremental models to save costs & speed 4️⃣ Consumption BI dashboards (Power BI / Tableau) connected to CURATED schema Stakeholders accessing real-time KPIs with confidence 🔹 Snowflake Features Used ✔ Time Travel – Rollback transformations when needed ✔ Streams + Tasks – Handle CDC for near real-time updates ✔ Materialized Views – For frequently accessed aggregations ✔ Zero-Copy Cloning – Testing new dbt models without extra cost 🔹 Impact Delivered 🔥 Reduced pipeline runtime by 60% 🔥 Improved trust with dbt tests (unique, not null, accepted values) 🔥 Delivered a data lineage graph for end-to-end transparency 🔥 Enabled self-service analytics for multiple business teams 💡 Takeaway: Snowflake provides the power, dbt provides the discipline. Together, they create pipelines that are scalable, auditable, and business-ready. 👉 Question for you: If you could only pick one feature to highlight — would you choose Snowflake’s ❄️ Time Travel or dbt’s 📦 Data Tests? #Snowflake #dbt #DataEngineering #ModernDataStack #ELT #SeniorDataEngineer #BigData #ETL #DataPipeline #CloudDataWarehouse

23

SQL Server 2025 (Preview) introduces vector columns and vector indexes enabling AI-powered similarity search directly in the database. In simple terms: SQL Server can now find results based on meaning, not just exact text matches. What are these? Vector columns: store AI embeddings (numeric representations of text, products, images, logs, etc.) Vector indexes: enable fast nearest-neighbor and similarity search Vector search: lets SQL Server answer queries like “show items similar to this” or “find related documents.” Why this matters for business? Build AI search, recommendation engines, or fraud detection directly on SQL Server data Leverage existing infrastructure, security, and governance - no new databases required Lower the barrier to adopting AI in production systems Some Practical considerations - Drawbacks - Embeddings and indexes can increase storage and memory usage - Approximate search may not suit critical regulatory or fraud queries (YET!) - Vector columns require fixed dimensions and embeddings generated externally - Possibly meaning that changes should be applied in the application level - For small datasets, vector indexes may not improve performance SQL Server 2025 makes AI integration practical — but DBAs should plan for resource usage and approximation limits. source: https://lnkd.in/d3GS85SB  

18

3 Comments

Most ELT pipelines are slow, fragile, and require niche skills just to stay running. Starburst simplifies the complexity with a high-performance, SQL-based engine that handles your longest queries—without forcing a migration. Build pipelines directly on your data lake—Leverage SQL, not Scala. And transform data at scale with full fault tolerance and flexibility. This is ELT done right. No extra tools. No extra complexity. 🔗 Learn how Starburst accelerates modern data processing: https://okt.to/dlD1vu #ELT #DataProcessing #SQLatScale #DataPipelines #Starburst #Trino #DataEngineering #ModernDataStack

20

1 Comment

Data bootcamps teach you Python, SQL, and machine learning. But they don't teach you the most important skill. How to smell when something's wrong with your data.  How to know which messy parts to keep and which clean parts to question.  How to understand the business process that created your dataset.  How to find the human behind every weird pattern. These skills matter more than knowing every algorithm. You can have the fanciest model in the world, but if your data understanding is wrong, your insights will be useless.  Learn to read the mess, not just clean it. Read the full post here: https://lnkd.in/e3D7fmiC

3

4 Comments

Ever wondered what it takes to switch from software development to a data expert? In the latest episode of the sql_squared podcast, I had a fascinating conversation with the highly experienced Simon Frazer, a Senior DBA and SQL Server consultant. We explored his incredible career journey, including: 🔹 The key differences (and mindset shift) between being a dev and a DBA. 🔹 Real-world performance tuning tips from his time at a major video game retailer. 🔹 How he uses tools like the Brent Ozar Toolkit to solve complex problems. 🔹 The value of getting involved in the data community. A massive thank you to Simon Frazer for sharing his knowledge and deep insights with us. It was a fantastic discussion! 🎧 Listen to "Evolving Roles: A Journey to Data Expert" now! Youtube: https://lnkd.in/egsj8uA2 Web: https://lnkd.in/euKi2bik #sql_squared #SQLServer #DBA #CareerJourney #PerformanceTuning #DataCommunity #Podcast #Tech

6