[Jul 21, 2026] Latest Databricks Certification Databricks-Certified-Professional-Data-Engineer Actual Free Exam Questions
Databricks Certification Databricks-Certified-Professional-Data-Engineer Dumps Updated Practice Test and 217 unique questions
Databricks Certified Professional Data Engineer (Databricks-Certified-Professional-Data-Engineer) certification exam is designed for data professionals who want to validate their skills and knowledge in building and deploying data engineering solutions using Databricks. Databricks is a unified data analytics platform that provides a collaborative environment for data engineers, data scientists, and business analysts to work together on big data projects. Databricks Certified Professional Data Engineer Exam certification exam covers a range of topics such as data ingestion, data processing, data transformation, and data storage using Databricks.
NEW QUESTION # 58
An external object storage container has been mounted to the location /mnt/finance_eda_bucket.
The following logic was executed to create a database for the finance team:
After the database was successfully created and permissions configured, a member of the finance team runs the following code:
If all users on the finance team are members of the finance group, which statement describes how the tx_sales table will be created?
- A. An managed table will be created in the storage container mounted to /mnt/finance eda bucket.
- B. An external table will be created in the storage container mounted to /mnt/finance eda bucket.
- C. A logical table will persist the query plan to the Hive Metastore in the Databricks control plane.
- D. A managed table will be created in the DBFS root storage container.
- E. A logical table will persist the physical plan to the Hive Metastore in the Databricks control plane.
Answer: A
Explanation:
https://docs.databricks.com/en/lakehouse/data-objects.html
NEW QUESTION # 59
What is the type of table created when you issue SQL DDL command CREATE TABLE sales (id int, units int)
- A. Query fails due to missing location
- B. Managed Parquet table
- C. Query fails due to missing format
- D. Managed Delta table
- E. External Table
Answer: D
Explanation:
Explanation
Answer is Managed Delta table
Anytime a table is created without the Location keyword it is considered a managed table, by de-fault all managed tables DELTA tables Syntax CREATE TABLE table_name ( column column_data_type...)
NEW QUESTION # 60
A data team ' s Structured Streaming job is configured to calculate running aggregates for item sales to update a downstream marketing dashboard. The marketing team has introduced a new field to track the number of times this promotion code is used for each item. A junior data engineer suggests updating the existing query as follows: Note that proposed changes are in bold.
Which step must also be completed to put the proposed query into production?
- A. Run REFRESH TABLE delta, /item_agg '
- B. Specify a new checkpointlocation
- C. Increase the shuffle partitions to account for additional aggregates
- D. Remove .option (mergeSchema ' , true ' ) from the streaming write
Answer: B
Explanation:
When introducing a new aggregation or a change in the logic of a Structured Streaming query, it is generally necessary to specify a new checkpoint location. This is because the checkpoint directory contains metadata about the offsets and the state of the aggregations of a streaming query. If the logic of the query changes, such as including a new aggregation field, the state information saved in the current checkpoint would not be compatible with the new logic, potentially leading to incorrect results or failures. Therefore, to accommodate the new field and ensure the streaming job has the correct starting point and state information for aggregations, a new checkpoint location should be specified.
:
Databricks documentation on Structured Streaming: https://docs.databricks.com/spark/latest/structured- streaming/index.html Databricks documentation on streaming checkpoints: https://docs.databricks.com/spark/latest/structured- streaming/production.html#checkpointing
NEW QUESTION # 61
A junior developer complains that the code in their notebook isn't producing the correct results in the development environment. A shared screenshot reveals that while they're using a notebook versioned with Databricks Repos, they're using a personal branch that contains old logic. The desired branch nameddev-2.3.9is not available from the branch selection dropdown.
Which approach will allow this developer to review the current logic for this notebook?
- A. Use Repos to merge the current branch and the dev-2.3.9 branch, then make a pull request to sync with the remote repository
- B. Merge all changes back to the main branch in the remote Git repository and clone the repo again
- C. Use Repos to make a pull request use the Databricks REST API to update the current branch to dev-2.3.9
- D. Use Repos to pull changes from the remote Git repository and select the dev-2.3.9 branch.
- E. Use Repos to checkout the dev-2.3.9 branch and auto-resolve conflicts with the current branch
Answer: D
Explanation:
Explanation
This is the correct answer because it will allow the developer to update their local repository with the latest changes from the remote repository and switch to the desired branch. Pulling changes will not affect the current branch or create any conflicts, as it will only fetch the changes and not merge them. Selecting the dev-2.3.9 branch from the dropdown will checkout that branch and display its contents in the notebook.
Verified References: [Databricks Certified Data Engineer Professional], under "Databricks Tooling" section; Databricks Documentation, under "Pull changes from a remote repository" section.
NEW QUESTION # 62
A data engineer is creating a data ingestion pipeline to understand where customers are taking their rented bicycles during use. The engineer noticed that over time, data being transmitted from the bicycle sensors fails to include key details like latitude and longitude. Downstream analysts need both the clean records and the quarantined records available for separate processing.
The data engineer already has this code:
import dlt
from pyspark.sql.functions import expr
rules = {
" valid_lat " : " (lat IS NOT NULL) " ,
" valid_long " : " (long IS NOT NULL) "
}
quarantine_rules = " NOT({0}) " .format( " AND " .join(rules.values()))
@dlt.view
def raw_trips_data():
return spark.readStream.table( " ride_and_go.telemetry.trips " )
How should the data engineer meet the requirements to capture good and bad data?
- A. @dlt.table
@dlt.expect_all_or_drop(rules)
def trips_data_quarantine():
return spark.readStream.table( " raw_trips_data " ) - B. @dlt.table(name= " trips_data_quarantine " )
def trips_data_quarantine():
return (
spark.readStream.table( " raw_trips_data " )
filter(expr(quarantine_rules))
) - C. @dlt.view
@dlt.expect_or_drop( " lat_long_present " , " (lat IS NOT NULL AND long IS NOT NULL) " ) def trips_data_quarantine():
return spark.readStream.table( " ride_and_go.telemetry.trips " ) - D. @dlt.table(partition_cols=[ " is_quarantined " ])
@dlt.expect_all(rules)
def trips_data_quarantine():
return (
spark.readStream.table( " raw_trips_data " )
withColumn( " is_quarantined " , expr(quarantine_rules))
)
Answer: D
Explanation:
Databricks documents a quarantine pattern for Lakeflow Spark Declarative Pipelines in which you create a dataset containing both valid and invalid rows, add an is_quarantined flag based on your rule set, and then use that dataset for separate downstream processing paths. The documented pattern uses a boolean quarantine expression and preserves all rows while tracking quality metrics through expectations. ( Databricks Documentation ) Option A is the only choice that matches that documented design: it keeps all records, marks invalid rows with is_quarantined , and applies expectations to capture data quality metrics. Option B drops invalid rows, which fails the requirement to keep quarantined records available. Option C captures only bad rows and loses the clean path. Option D also drops invalid rows and therefore does not preserve both good and quarantined records for separate downstream use. ( Databricks Documentation )
======
NEW QUESTION # 63
A junior data engineer on your team has implemented the following code block.
The viewnew_eventscontains a batch of records with the same schema as theeventsDelta table.
Theevent_idfield serves as a unique key for this table.
When this query is executed, what will happen with new records that have the sameevent_idas an existing record?
- A. They are inserted.
- B. They are merged.
- C. They are deleted.
- D. They are ignored.
- E. They are updated.
Answer: D
Explanation:
Explanation
This is the correct answer because it describes what will happen with new records that have the same event_id as an existing record when the query is executed. The query uses the INSERT INTO command to append new records from the view new_events to the table events. However, the INSERT INTO command does not check for duplicate values in the primary key column (event_id) and does not perform any update or delete operations on existing records. Therefore, if there are new records that have the same event_id as an existing record, they will be ignored and not inserted into the table events. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Append data using INSERT INTO" section.
"If none of the WHEN MATCHED conditions evaluate to true for a source and target row pair that matches the merge_condition, then the target row is left unchanged."https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html#:~:text=If%20none%20o
NEW QUESTION # 64
You were asked to identify number of times a temperature sensor exceed threshold temperature (100.00) by each device, each row contains 5 readings collected every 5 minutes, fill in the blank with the appropriate functions.
Schema: deviceId INT, deviceTemp ARRAY<double>, dateTimeCollected TIMESTAMP
SELECT deviceId, __ (__ (__(deviceTemp], i -> i > 100.00)))
FROM devices
GROUP BY deviceId
- A. SUM, SIZE, FILTER
- B. SUM, SIZE, ARRAY_FILTER
- C. SUM, SIZE, ARRAY_CONTAINS
- D. SUM, COUNT, SIZE
- E. SUM, SIZE, SLICE
Answer: A
Explanation:
Explanation
FILER function can be used to filter an array based on an expression
SIZE function can be used to get size of an array
SUM is used to calculate to total by device
Diagram Description automatically generated
NEW QUESTION # 65
A data analyst has provided a data engineering team with the following Spark SQL query:
1.SELECT district,
2.avg(sales)
3.FROM store_sales_20220101
4.GROUP BY district;
The data analyst would like the data engineering team to run this query every day. The date at the end of the
table name (20220101) should automatically be replaced with the current date each time the query is run.
Which of the following approaches could be used by the data engineering team to efficiently auto-mate this
process?
- A. They could pass the table into PySpark and develop a robustly tested module on the existing query
- B. They could replace the string-formatted date in the table with a timestamp-formatted date
- C. They could manually replace the date within the table name with the current day's date
- D. They could request that the data analyst rewrites the query to be run less frequently
- E. They could wrap the query using PySpark and use Python's string variable system to automatically
update the table name
Answer: E
NEW QUESTION # 66
A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in a bronze table created with the property delta.enableChangeDataFeed = true. They plan to execute the following code as a daily job:
Which statement describes the execution and results of running the above query multiple times?
- A. Each time the job is executed, only those records that have been inserted or updated since the last execution will be appended to the target table giving the desired result.
- B. Each time the job is executed, newly updated records will be merged into the target table, overwriting previous values with the same primary keys.
- C. Each time the job is executed, the differences between the original and current versions are calculated; this may result in duplicate entries for some records.
- D. Each time the job is executed, the target table will be overwritten using the entire history of inserted or updated records, giving the desired result.
- E. Each time the job is executed, the entire available history of inserted or updated records will be appended to the target table, resulting in many duplicate entries.
Answer: E
Explanation:
Reading table's changes, captured by CDF, using spark.read means that you are reading them as a static source. So, each time you run the query, all table's changes (starting from the specified startingVersion) will be read.
NEW QUESTION # 67
A data ingestion task requires a one-TB JSON dataset to be written out to Parquet with a target part-file size of 512 MB. Because Parquet is being used instead of Delta Lake, built-in file-sizing features such as Auto-Optimize & Auto-Compaction cannot be used.
Which strategy will yield the best performance without shuffling data?
- A. Set spark.sql.files.maxPartitionBytes to 512 MB, ingest the data, execute the narrow transformations, and then write to parquet.
- B. Set spark.sql.shuffle.partitions to 512, ingest the data, execute the narrow transformations, and then write to parquet.
- C. Set spark.sql.adaptive.advisoryPartitionSizeInBytes to 512 MB bytes, ingest the data, execute the narrow transformations, coalesce to 2,048 partitions (1TB*1024*1024/512), and then write to parquet.
- D. Ingest the data, execute the narrow transformations, repartition to 2,048 partitions (1TB* 1024*1024/512), and then write to parquet.
- E. Set spark.sql.shuffle.partitions to 2,048 partitions (1TB*1024*1024/512), ingest the data, execute the narrow transformations, optimize the data by sorting it (which automatically repartitions the data), and then write to parquet.
Answer: A
Explanation:
For this scenario where a one-TB JSON dataset needs to be converted into Parquet format without employing Delta Lake's auto-sizing features, the goal is to avoid unnecessary data shuffles and yet ensure optimal file sizes for the output Parquet files. Here's a breakdown of why option A is most suitable:
Setting maxPartitionBytes: The spark.sql.files.maxPartitionBytes configuration controls the size of blocks that Spark reads from the data source (in this case, the JSON files) but also influences the output size of files when data is written without repartition or coalesce operations. Setting this parameter to 512 MB directly addresses the requirement to manage the output file size effectively.
Data Ingestion and Processing:
Ingesting Data: Load the JSON dataset into a DataFrame.
Applying Transformations: Perform any required narrow transformations that do not involve shuffling data (like filtering or adding new columns).
Writing to Parquet: Directly write the transformed DataFrame to Parquet files. The setting for maxPartitionBytes ensures that each part-file is approximately 512 MB, meeting the requirement for part-file size without additional steps to repartition or coalesce the data.
Performance Consideration: This approach is optimal because:
It avoids the overhead of shuffling data, which can be significant, especially with large datasets.
It directly ties the read/write operations to a configuration that matches the target output size, making it efficient in terms of both computation and I/O operations.
Alternative Options Analysis:
Option B and D: Involves repartitioning, which would trigger a shuffle of the data, contradicting the requirement to avoid shuffling for performance reasons.
Option C: Uses coalesce, which is less intensive than repartition but can still lead to uneven partition sizes and does not directly control the output file size as effectively as setting maxPartitionBytes.
Option E: Setting shuffle partitions to 512 doesn't directly control the output file size for writing to Parquet and could lead to smaller files depending on the dataset's partitioning post-transformations.
Reference
Apache Spark Configuration
Writing to Parquet Files in Spark
NEW QUESTION # 68
A junior data engineer has configured a workload that posts the following JSON to the Databricks REST API endpoint 2.0/jobs/create.
Assuming that all configurations and referenced resources are available, which statement describes the result of executing this workload three times?
- A. The logic defined in the referenced notebook will be executed three times on the referenced existing all purpose cluster.
- B. Three new jobs named "Ingest new data" will be defined in the workspace, and they will each run once daily.
- C. The logic defined in the referenced notebook will be executed three times on new clusters with the configurations of the provided cluster ID.
- D. One new job named "Ingest new data" will be defined in the workspace, but it will not be executed.
- E. Three new jobs named "Ingest new data" will be defined in the workspace, but no jobs will be executed.
Answer: E
Explanation:
This is the correct answer because the JSON posted to the Databricks REST API endpoint 2.0/jobs/create defines a new job with a name, an existing cluster id, and a notebook task. However, it does not specify any schedule or trigger for the job execution. Therefore, three new jobs with the same name and configuration will be created in the workspace, but none of them will be executed until they are manually triggered or scheduled. Verified Reference: [Databricks Certified Data Engineer Professional], under "Monitoring & Logging" section; [Databricks Documentation], under "Jobs API - Create" section.
NEW QUESTION # 69
Which statement regarding stream-static joins and static Delta tables is correct?
- A. Each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch.
- B. The checkpoint directory will be used to track updates to the static Delta table.
- C. The checkpoint directory will be used to track state information for the unique keys present in the join.
- D. Each microbatch of a stream-static join will use the most recent version of the static Delta table as of the job's initialization.
- E. Stream-static joins cannot use static Delta tables because of consistency issues.
Answer: A
Explanation:
This is the correct answer because stream-static joins are supported by Structured Streaming when one of the tables is a static Delta table. A static Delta table is a Delta table that is not updated by any concurrent writes, such as appends or merges, during the execution of a streaming query. In this case, each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch, which means it will reflect any changes made to the static Delta table before the start of each microbatch. Verified References: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "Stream and static joins" section.
NEW QUESTION # 70
A data engineer inherits a Delta table with historical partitions by country that are badly skewed. Queries often filter by high-cardinality customer_id and vary across dimensions over time. The engineer wants a strategy that avoids a disruptive full rewrite, reduces sensitivity to skewed partitions, and sustains strong query performance as access patterns evolve.
Which two actions should the data engineer take? (Choose 2)
- A. Switch from static partitioning to liquid clustering and select initial clustering keys that reflect common filters such as customer_id.
- B. Keep existing partitions and rely on bin-packing OPTIMIZE only; ZORDER and clustering are unnecessary for multi-dimensional filters.
- C. Periodically run OPTIMIZE table_name.
- D. Depend solely on optimized writes; Databricks will automatically replace partitioning with clustering over time.
- E. Disable data skipping statistics to avoid maintenance overhead; rely on adaptive query execution instead.
Answer: A,C
Explanation:
Liquid Clustering replaces traditional partitioning and ZORDER optimization by automatically organizing data according to clustering keys. It supports evolving clustering strategies without requiring a full table rewrite. To maintain cluster balance and improve performance, the OPTIMIZE command should be run periodically. OPTIMIZE groups data files by clustering keys and helps reduce small file overhead.
Reference Source: Databricks Delta Lake Guide - "Use Liquid Clustering for Tables" and "OPTIMIZE Command for File Compaction and Data Layout."
NEW QUESTION # 71
A data engineer wants to create a cluster using the Databricks CLI for a big ETL pipeline. The cluster should have five workers, one driver of type i3.xlarge, and should use the '14.3.x-scala2.12' runtime.
Which command should the data engineer use?
- A. databricks compute create 14.3.x-scala2.12 --num-workers 5 --node-type-id i3.xlarge --cluster-name Data Engineer_cluster
- B. databricks clusters create 14.3.x-scala2.12 --num-workers 5 --node-type-id i3.xlarge --cluster-name DataEngineer_cluster
- C. databricks clusters add 14.3.x-scala2.12 --num-workers 5 --node-type-id i3.xlarge --cluster-name Data Engineer_cluster
- D. databricks compute add 14.3.x-scala2.12 --num-workers 5 --node-type-id i3.xlarge --cluster-name Data Engineer_cluster
Answer: A
Explanation:
The Databricks CLI allows users to manage clusters using command-line commands. The correct command for creating a cluster follows a specific format.
Key Components in the Command:
Command Type: databricks compute create is the correct syntax for creating a new compute resource (cluster).
Runtime Version: '14.3.x-scala2.12' specifies the Databricks runtime to use.
Workers: --num-workers 5 sets the number of worker nodes to 5.
Node Type: --node-type-id i3.xlarge defines the hardware configuration.
Cluster Name: --cluster-name DataEngineer_cluster assigns a recognizable name to the cluster.
Evaluation of Options:
Option A (databricks clusters create ...)
Incorrect: databricks clusters create is not a valid command in the Databricks CLI v0.205.
The correct CLI command for cluster creation is databricks compute create.
Option B (databricks clusters add ...)
Incorrect: databricks clusters add is not a valid CLI command.
Option C (databricks compute add ...)
Incorrect: databricks compute add is not a valid CLI command.
Option D (databricks compute create ...)
Correct: databricks compute create is the correct command for creating a cluster.
Conclusion:
The correct command to create a cluster with five workers, an i3.xlarge node type, and Databricks runtime 14.3.x-scala2.12 is:
databricks compute create 14.3.x-scala2.12 --num-workers 5 --node-type-id i3.xlarge --cluster-name Data Engineer_cluster Thus, the correct answer is D.
Reference:
Databricks CLI Documentation
NEW QUESTION # 72
Which approach demonstrates a modular and testable way to use DataFrame.transform for ETL code in PySpark?
- A. def upper_transform(df):
return df.withColumn( " value_upper " , upper(col( " value " )))
actual = test_input.transform(upper_transform)
assertDataFrameEqual(actual, expected) - B. class Pipeline:
def transform(self, df):
return df.withColumn( " value_upper " , upper(col( " value " )))
pipeline = Pipeline()
assertDataFrameEqual(pipeline.transform(test_input), expected) - C. def transform_data(input_df):
# transformation logic here
return output_df
test_input = spark.createDataFrame([(1, " a " )], [ " id " , " value " ]) assertDataFrameEqual(transform_data(test_input), expected) - D. def upper_value(df):
return df.withColumn( " value_upper " , upper(col( " value " )))
def filter_positive(df):
return df.filter(df[ " id " ] > 0)
pipeline_df = df.transform(upper_value).transform(filter_positive)
Answer: D
Explanation:
Databricks and Apache Spark recommend building modular and reusable ETL transformations by leveraging the DataFrame.transform() API. This method allows you to chain multiple transformation functions in a clean and testable way.
* Option A : Encapsulating the logic in a class (Pipeline) works, but it reduces modularity and flexibility.
It does not show the true intended use of DataFrame.transform() which is chaining functional transformations.
* Option B : This is the correct approach. It defines small, reusable functions (upper_value, filter_positive) that each take a DataFrame and return a transformed DataFrame. By chaining them with df.transform(func), you can compose ETL pipelines in a clear and declarative manner. This enables unit testing of individual functions and makes the ETL pipeline modular, testable, and production- ready.
* Option C : This shows a single transformation wrapped in a function and tested, but it lacks pipeline composition - it is not demonstrating modular chaining across multiple transformations.
* Option D : This simply defines a transformation function with hardcoded logic. It does not leverage DataFrame.transform() nor demonstrate modularity through composition.
Therefore, Option B is the best demonstration of how to use DataFrame.transform() in PySpark ETL pipelines.
Databricks documentation explicitly highlights that DataFrame.transform() allows developers to "chain together reusable functions in a readable and modular way, improving testability and maintainability of ETL code." This makes B the correct and officially supported pattern.
NEW QUESTION # 73
Unity catalog simplifies managing multiple workspaces, by storing and managing permissions and ACL at
_______ level
- A. Account
(Correct) - B. Data pane
- C. Storage
- D. Workspace
- E. Control pane
Answer: A
Explanation:
Explanation
The answer is, Account Level
The classic access control list (tables, workspace, cluster) is at the workspace level, Unity catalog is at the account level and can manage all the workspaces in an Account.
NEW QUESTION # 74
A table is registered with the following code:
Bothusersandordersare Delta Lake tables. Which statement describes the results of queryingrecent_orders?
- A. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query finishes.
- B. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query began.
- C. The versions of each source table will be stored in the table transaction log; query results will be saved to DBFS with each query.
- D. Results will be computed and cached when the table is defined; these cached results will incrementally update as new records are inserted into source tables.
- E. All logic will execute when the table is definedand store the result of joiningtables to the DBFS; this stored data will be returned when the table is queried.
Answer: B
Explanation:
Explanation
This is the correct answer because Delta Lake supports time travel, which allows users to query data as of a specific version or timestamp. The code uses the VERSION AS OF syntax to specify the version of each source table to be used in the join. The result of querying recent_orders will be the same as joining those versions of the source tables at query time. The query will use snapshot isolation, which means it will use a consistent snapshot of the table at the time the query began, regardless of any concurrent updates or deletes.
Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Query an older snapshot of a table (time travel)" section.
NEW QUESTION # 75
A Delta Lake table was created with the below query:
Realizing that the original query had a typographical error, the below code was executed:
ALTER TABLE prod.sales_by_stor RENAME TO prod.sales_by_store
Which result will occur after running the second command?
- A. The table reference in the metastore is updated and all data files are moved.
- B. All related files and metadata are dropped and recreated in a single ACID transaction.
- C. The table reference in the metastore is updated and no data is changed.
- D. The table name change is recorded in the Delta transaction log.
- E. A new Delta transaction log Is created for the renamed table.
Answer: C
Explanation:
The query uses the CREATE TABLE USING DELTA syntax to create a Delta Lake table from an existing Parquet file stored in DBFS. The query also uses the LOCATION keyword to specify the path to the Parquet file as /mnt/finance_eda_bucket/tx_sales.parquet. By using the LOCATION keyword, the query creates an external table, which is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An externaltable can be created from an existing directory in a cloud storage system, such as DBFS or S3, that contains data files in a supported format, such as Parquet or CSV.
The result that will occur after running the second command is that the table reference in the metastore is updated and no data is changed. The metastore is a service that stores metadata about tables, such as their schema, location, properties, and partitions. The metastore allows users to access tables using SQL commands or Spark APIs without knowing their physical location or format. When renaming an external table using the ALTER TABLE RENAME TO command, only the table reference in the metastore is updated with the new name; no data files or directories are moved or changed in the storage system. The table will still point to the same location and use the same format as before. However, if renaming a managed table, which is a table whose metadata and data are both managed by Databricks, both the table reference in the metastore and the data files in the default warehouse directory are moved and renamed accordingly. Verified References:
[Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "ALTER TABLE RENAME TO" section; Databricks Documentation, under "Metastore" section; Databricks Documentation, under "Managed and external tables" section.
NEW QUESTION # 76
Review the following error traceback:
Which statement describes the error being raised?
- A. The code executed was PvSoark but was executed in a Scala notebook.
- B. There is a type error because a DataFrame object cannot be multiplied.
- C. There is a syntax error because the heartrate column is not correctly identified as a column.
- D. There is no column in the table named heartrateheartrateheartrate
- E. There is a type error because a column object cannot be multiplied.
Answer: D
Explanation:
The error being raised is an AnalysisException, which is a type of exception that occurs when Spark SQL cannot analyze or execute a query due to some logical or semantic error 1 . In this case, the error message indicates that the query cannot resolve the column name 'heartrateheartrateheartrate' given the input columns
'heartrate' and 'age'. This means that there is no column in the table named 'heartrateheartrateheartrate', and the query is invalid. A possible cause of this error is a typo or a copy-paste mistake in the query. To fix this error, the query should use a valid column name that exists in the table, such as
'heartrate'. References: AnalysisException
NEW QUESTION # 77
Which of the following is correct for the global temporary view?
- A. global temporary views can be accessed across many clusters
- B. global temporary views are created in a database called temp database
- C. global temporary views can be still accessed even if the cluster is restarted
- D. global temporary views can be still accessed even if the notebook is detached and at-tached
- E. global temporary views cannot be accessed once the notebook is detached and attached
Answer: D
Explanation:
Explanation
The answer is global temporary views can be still accessed even if the notebook is detached and attached There are two types of temporary views that can be created Local and Global
* A local temporary view is only available with a spark session, so another notebook in the same cluster can not access it. if a notebook is detached and reattached local temporary view is lost.
* A global temporary view is available to all the notebooks in the cluster, even if the notebook is detached and reattached it can still be accessible but if a cluster is restarted the global temporary view is lost.
NEW QUESTION # 78
Which of the following technologies can be used to identify key areas of text when parsing Spark Driver log4j output?
- A. Regex
- B. Julia
- C. Scala Datasets
- D. pyspsark.ml.feature
- E. C++
Answer: A
Explanation:
Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks. References:
* https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions
* https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html
NEW QUESTION # 79
A Structured Streaming job deployed to production has been resulting in higher than expected cloud storage costs. At present, during normal execution, each micro-batch of data is processed in less than 3 seconds; at least 12 times per minute, a micro-batch is processed that contains 0 records. The streaming write was configured using the default trigger settings. The production job is currently scheduled alongside many other Databricks jobs in a workspace with instance pools provisioned to reduce start-up time for jobs with batch execution. Holding all other variables constant and assuming records need to be processed in less than 10 minutes, which adjustment will meet the requirement?
- A. Set the trigger interval to 10 minutes; each batch calls APIs in the source storage account, so decreasing trigger frequency to the maximum allowable threshold should minimize this cost.
- B. Use the trigger once option and configure a Databricks job to execute the query every 10 minutes; this approach minimizes costs for both compute and storage.
- C. Set the trigger interval to 500 milliseconds; setting a small but non-zero trigger interval ensures that the source is not queried too frequently.
- D. Set the trigger interval to 3 seconds; the default trigger interval is consuming too many records per batch, resulting in spill to disk that can increase volume costs.
Answer: B
Explanation:
Comprehensive and Detailed Explanation From Exact Extract:
* Exact extract: "If no trigger is specified, the default processing-time trigger runs micro-batches as fast as possible."
* Exact extract: "Trigger once processes all available data once and then stops." References: Structured Streaming triggers; Databricks Jobs and job clusters.
NEW QUESTION # 80
The business intelligence team has a dashboard configured to track various summary metrics for retail stories.
This includes total sales for the previous day alongside totals and averages for a variety of time periods. The fields required to populate this dashboard have the following schema:
For Demand forecasting, the Lakehouse contains a validated table of all itemized sales updated incrementally in near real-time. This table named products_per_order, includes the following fields:
Because reporting on long-term sales trends is less volatile, analysts using the new dashboard only require data to be refreshed once daily. Because the dashboard will be queried interactively by many users throughout a normal business day, it should return results quickly and reduce total compute associated with each materialization.
Which solution meets the expectations of the end users while controlling and limiting possible costs?
- A. Use Structure Streaming to configure a live dashboard against the products_per_order table within a Databricks notebook.
- B. Populate the dashboard by configuring a nightly batch job to save the required to quickly update the dashboard with each query.
- C. Use the Delta Cache to persists the products_per_order table in memory to quickly the dashboard with each query.
- D. Define a view against the products_per_order table and define the dashboard against this view.
Answer: D
Explanation:
Given the requirement for daily refresh of data and the need to ensure quick response times for interactive queries while controlling costs, a nightly batch job to pre-compute and save the required summary metrics is the most suitable approach.
* By pre-aggregating data during off-peak hours, the dashboard can serve queries quickly without requiring on-the-fly computation, which can be resource-intensive and slow, especially with many users.
* This approach also limits the cost by avoiding continuous computation throughout the day and instead leverages a batch process that efficiently computes and stores the necessary data.
* The other options (A, C, D) either do not address the cost and performance requirements effectively or
* are not suitable for the use case of less frequent data refresh and high interactivity.
References:
* Databricks Documentation on Batch Processing: Databricks Batch Processing
* Data Lakehouse Patterns: Data Lakehouse Best Practices
NEW QUESTION # 81
......
Verified Databricks-Certified-Professional-Data-Engineer dumps Q&As - 100% Pass from DumpsQuestion: https://www.dumpsquestion.com/Databricks-Certified-Professional-Data-Engineer-exam-dumps-collection.html