Prove your data engineering skills
Data engineering is judged by what happens when things go wrong: late files, duplicate events, a schema that changed overnight. Lemma records how you design for those failures, through live sessions, assessments and code you connect.
What data engineering competence looks like
A competent data engineer builds pipelines that can run twice without doing damage. They make jobs idempotent, partition data so a single day can be reprocessed, and decide clearly between batch and streaming based on what consumers need. They model data so analysts can query it without reverse engineering the source systems.
They also plan for change and failure. They validate schemas and data quality at the edges, alert on freshness and volume rather than only on job failures, and keep lineage clear enough to answer where a number came from. Cost is part of the design, not a surprise on the invoice.
Designing a pipeline under constraints
60 min · Learning · with Tomás G.
Tomás G. rated the session
Practitioner · after the call
Clear reasoning on pipeline design. Talked through trade-offs first.
Pipeline design
Idempotent jobs, backfills and partitioning that make reruns and recovery safe and routine.
Batch and streaming
Choosing between them, handling late and duplicate events, and understanding delivery guarantees.
Data modelling
Dimensional models, slowly changing dimensions and tables that analysts can query with confidence.
Orchestration
Scheduling, dependencies, retries and alerting with tools such as Airflow or Dagster.
Data quality and contracts
Schema validation, tests on freshness and volume, and agreements with upstream producers.
Storage and cost
Warehouses, lakes, file formats and the query patterns that drive storage and compute costs.
What counts as evidence of data engineering skills?
In data engineering, a polished result can hide a shaky method. Lemma records the evidence behind the method: sessions where someone followed your reasoning, an adaptive test and the repositories you link, notebooks included. Each source is listed below. See the evidence methodology for how each source is weighed.
Data engineering
Sven N. · SCI recomputed 9 Oct
Evidence · 4 entries
- Rated 5.0
Designing a pipeline under constraints
Live session · 9 Oct
- Verified
Pipeline code on GitHub
GitHub connector · 2 Oct
- Level 7.2
Debugging a broken transformation
Adaptive assessment · 24 Sep
- Rated 5.0
Incident walk-through
Live session · 11 Sep
Designing a pipeline under constraints
A practitioner describes the sources, rough volumes and the people who consume the data. In a live session you design the pipeline and explain how it handles late data, reruns and schema changes, and they rate it.
Pipeline code on GitHub
Connect a repository with DAGs, transformation models or ingestion code you wrote. Lemma checks the repository exists and that your account has access to it.
Debugging a broken transformation
Adaptive scenario questions cover modelling, orchestration and quality. At higher difficulty you fix broken pipeline or transformation code and explain the failure, scored against a rubric.
Incident walk-through
Walk a practitioner through a pipeline failure you handled: what broke, how you found it and what you changed. Their rating of your explanation is recorded as evidence.
How is a data engineering skills test run on Lemma?
The data engineering test asks 15 to 25 questions and moves up or down a level with each answer, focusing on judgement calls such as leakage, skew and bad joins. It feeds your data engineering SCI together with sessions and linked work, and the score weighs how fresh that evidence is.
Rerunning a failed daily job doubles yesterday's rows. Which design prevents this?
- AIdempotent loads that overwrite by partition
- BRun the job twice as often
- CAdd more workers
- DTurn off retries
Practise core patterns
The AI tutor sets exercises on idempotency, partitioning, modelling and SQL transformations, adapting as you answer. Drills are preparation and add only a small, capped amount to your practice score.
Design with a practitioner
In live sessions, design or review pipelines with someone who builds them. Completion is recorded when you both join, and the practitioner's rating becomes data engineering evidence.
Complete the assessment
Scenario questions on pipelines, modelling and quality adapt to your level, with debug challenges as difficulty rises. Per-question time limits and no back-navigation keep the result honest.
Data engineering skills: common questions
How do I move from data analyst to data engineer and prove it?
What do data engineering interviews focus on?
Which tools does the data engineering skill cover?
Can a personal project count as data engineering evidence?
Skills related to data engineering
Browse the skills directorySQL
Prove you can write correct, fast queries on real schemas, from joins to window functions and query plans.
Evidence for SQLPython
Show how you write, debug and structure Python, not just that you once finished a course.
Evidence for PythonCloud architecture
Show how you design cloud systems that scale, fail gracefully and stay affordable.
Evidence for cloud architectureMachine learning
Show how you frame, train, evaluate and debug models, with evidence a practitioner has reviewed live.
Evidence for machine learning
Start your data engineering evidence record.
Lemma is in early access. Request an invite and we will email you when your place is ready.