Databricks-Generative-AI-Engineer-Associate exam questions

1. Question 1

   Multiple answer

   A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author’s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user’s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.

   Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

   • A

     Change embedding models and compare performance.

   • B

     Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.

   • C

     Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters.Choose the strategy that gives the best performance metric.

   • D

     Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.

   • E

     Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.

   Reveal answer

   Correct answer

   • C

     Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters.Choose the strategy that gives the best performance metric.

   • E

     Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.

   Explanation

   To optimize a chunking strategy for a Retrieval-Augmented Generation (RAG) application, the Generative AI Engineer needs a structured approach to evaluating the chunking strategy, ensuring that the chosen configuration retrieves the most relevant information and leads to accurate and coherent LLM responses. Here's why C and E are the correct strategies:

   Strategy C: Evaluation Metrics (Recall, NDCG)

   Define an evaluation metric: Common evaluation metrics such as recall, precision, or NDCG (Normalized Discounted Cumulative Gain) measure how well the retrieved chunks match the user's query and the expected response.

   Recall measures the proportion of relevant information retrieved.

   NDCG is often used when you want to account for both the relevance of retrieved chunks and the ranking or order in which they are retrieved.

   Experiment with chunking strategies: Adjusting chunking strategies based on text structure (e.g., splitting by paragraph, chapter, or a fixed number of tokens) allows the engineer to experiment with various ways of slicing the text. Some chunks may better align with the user's query than others.

   Evaluate performance: By using recall or NDCG, the engineer can methodically test various chunking strategies to identify which one yields the highest performance. This ensures that the chunking method provides the most relevant information when embedding and retrieving data from the vector store.

   Strategy E: LLM-as-a-Judge Metric

   Use the LLM as an evaluator: After retrieving chunks, the LLM can be used to evaluate the quality of answers based on the chunks provided. This could be framed as a "judge" function, where the LLM compares how well a given chunk answers previous user queries.

   Optimize based on the LLM's judgment: By having the LLM assess previous answers and rate their relevance and accuracy, the engineer can collect feedback on how well different chunking configurations perform in real-world scenarios.

   This metric could be a qualitative judgment on how closely the retrieved information matches the user's intent.

   Tune chunking parameters: Based on the LLM's judgment, the engineer can adjust the chunk size or structure to better align with the LLM's responses, optimizing retrieval for future queries.

   By combining these two approaches, the engineer ensures that the chunking strategy is systematically evaluated using both quantitative (recall/NDCG) and qualitative (LLM judgment) methods. This balanced optimization process results in improved retrieval relevance and, consequently, better response generation by the LLM.

2. Question 2

   Single answer

   A Generative AI Engineer is designing a RAG application for answering user questions on technical regulations as they learn a new sport.

   What are the steps needed to build this RAG application and deploy it?

   • A

     Ingest documents from a source –> Index the documents and saves to Vector Search –> User submits queries against an LLM –> LLM retrieves relevant documents –> Evaluate model –> LLM generates a response –> Deploy it using Model Serving

   • B

     Ingest documents from a source –> Index the documents and save to Vector Search –> User submits queries against an LLM –> LLM retrieves relevant documents –> LLM generates a response -> Evaluate model –> Deploy it using Model Serving

   • C

     Ingest documents from a source –> Index the documents and save to Vector Search –> Evaluate model –> Deploy it using Model Serving

   • D

     User submits queries against an LLM –> Ingest documents from a source –> Index the documents and save to Vector Search –> LLM retrieves relevant documents –> LLM generates a response –> Evaluate model –> Deploy it using Model Serving

   Reveal answer

   Correct answer

   • B

     Ingest documents from a source –> Index the documents and save to Vector Search –> User submits queries against an LLM –> LLM retrieves relevant documents –> LLM generates a response -> Evaluate model –> Deploy it using Model Serving

   Explanation

   The Generative AI Engineer needs to follow a methodical pipeline to build and deploy a Retrieval-Augmented Generation (RAG) application. The steps outlined in option B accurately reflect this process:

   Ingest documents from a source: This is the first step, where the engineer collects documents (e.g., technical regulations) that will be used for retrieval when the application answers user questions.

   Index the documents and save to Vector Search: Once the documents are ingested, they need to be embedded using a technique like embeddings (e.g., with a pre-trained model like BERT) and stored in a vector database (such as Pinecone or FAISS). This enables fast retrieval based on user queries.

   User submits queries against an LLM: Users interact with the application by submitting their queries. These queries will be passed to the LLM.

   LLM retrieves relevant documents: The LLM works with the vector store to retrieve the most relevant documents based on their vector representations.

   LLM generates a response: Using the retrieved documents, the LLM generates a response that is tailored to the user's question.

   Evaluate model: After generating responses, the system must be evaluated to ensure the retrieved documents are relevant and the generated response is accurate. Metrics such as accuracy, relevance, and user satisfaction can be used for evaluation.

   Deploy it using Model Serving: Once the RAG pipeline is ready and evaluated, it is deployed using a model-serving platform such as Databricks Model Serving. This enables real-time inference and response generation for users.

   By following these steps, the Generative AI Engineer ensures that the RAG application is both efficient and effective for the task of answering technical regulation questions.

3. Question 3

   Single answer

   A Generative Al Engineer is tasked with improving the RAG quality by addressing its inflammatory outputs.

   Which action would be most effective in mitigating the problem of offensive text outputs?

   • A

     Increase the frequency of upstream data updates

   • B

     Inform the user of the expected RAG behavior

   • C

     Restrict access to the data sources to a limited number of users

   • D

     Curate upstream data properly that includes manual review before it is fed into the RAG system

   Reveal answer

   Correct answer

   • D

     Curate upstream data properly that includes manual review before it is fed into the RAG system

   Explanation

   Addressing offensive or inflammatory outputs in a Retrieval-Augmented Generation (RAG) system is critical for improving user experience and ensuring ethical AI deployment. Here's why D is the most effective approach:

   Manual data curation: The root cause of offensive outputs often comes from the underlying data used to train the model or populate the retrieval system. By manually curating the upstream data and conducting thorough reviews before the data is fed into the RAG system, the engineer can filter out harmful, offensive, or inappropriate content.

   Improving data quality: Curating data ensures the system retrieves and generates responses from a high-quality, well-vetted dataset. This directly impacts the relevance and appropriateness of the outputs from the RAG system, preventing inflammatory content from being included in responses.

   Effectiveness: This strategy directly tackles the problem at its source (the data) rather than just mitigating the consequences (such as informing users or restricting access). It ensures that the system consistently provides non-offensive, relevant information.

   Other options, such as increasing the frequency of data updates or informing users about behavior expectations, may not directly mitigate the generation of inflammatory outputs.

4. Getting the hang of it? 85 more Databricks-Generative-AI-Engineer-Associate questions are waiting, each with its answer. Unlock them all

5. Question 4

   Single answer

   A Generative Al Engineer is creating an LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative Al Engineer knows that cost and latency are more important than quality for this application. They have several context length levels to choose from.

   Which will fulfill their need?

   • A

     context length 514; smallest model is 0.44GB and embedding dimension 768

   • B

     context length 2048: smallest model is 11GB and embedding dimension 2560

   • C

     context length 32768: smallest model is 14GB and embedding dimension 4096

   • D

     context length 512: smallest model is 0.13GB and embedding dimension 384

   Reveal answer

   Correct answer

   • D

     context length 512: smallest model is 0.13GB and embedding dimension 384

   Explanation

   When prioritizing cost and latency over quality in a Large Language Model (LLM)-based application, it is crucial to select a configuration that minimizes both computational resources and latency while still providing reasonable performance. Here's why D is the best choice:

   Context length: The context length of 512 tokens aligns with the chunk size used for the documents (maximum of 512 tokens per chunk). This is sufficient for capturing the needed information and generating responses without unnecessary overhead.

   Smallest model size: The model with a size of 0.13GB is significantly smaller than the other options. This small footprint ensures faster inference times and lower memory usage, which directly reduces both latency and cost.

   Embedding dimension: While the embedding dimension of 384 is smaller than the other options, it is still adequate for tasks where cost and speed are more important than precision and depth of understanding.

   This setup achieves the desired balance between cost-efficiency and reasonable performance in a latency-sensitive, cost-conscious application.

6. Question 5

   Single answer

   A small and cost-conscious startup in the cancer research field wants to build a RAG application using Foundation Model APIs.

   Which strategy would allow the startup to build a good-quality RAG application while being cost-conscious and able to cater to customer needs?

   • A

     Limit the number of relevant documents available for the RAG application to retrieve from

   • B

     Pick a smaller LLM that is domain-specific

   • C

     Limit the number of queries a customer can send per day

   • D

     Use the largest LLM possible because that gives the best performance for any general queries

   Reveal answer

   Correct answer

   • B

     Pick a smaller LLM that is domain-specific

   Explanation

   For a small, cost-conscious startup in the cancer research field, choosing a domain-specific and smaller LLM is the most effective strategy. Here's why B is the best choice:

   Domain-specific performance: A smaller LLM that has been fine-tuned for the domain of cancer research will outperform a general-purpose LLM for specialized queries. This ensures high-quality responses without needing to rely on a large, expensive LLM.

   Cost-efficiency: Smaller models are cheaper to run, both in terms of compute resources and API usage costs. A domain-specific smaller LLM can deliver good quality responses without the need for the extensive computational power required by larger models.

   Focused knowledge: In a specialized field like cancer research, having an LLM tailored to the subject matter provides better relevance and accuracy for queries, while keeping costs low. Large, general-purpose LLMs may provide irrelevant information, leading to inefficiency and higher costs.

   This approach allows the startup to balance quality, cost, and customer satisfaction effectively, making it the most suitable strategy.

7. Question 6 Locked Unlock to see this question
8. Question 7 Locked Unlock to see this question

Back to the Databricks-Generative-AI-Engineer-Associate exam page