UMD Virtual Agent is an offering by the AI Solutions team that provides GenAI chatbots using Retrieval-Augmented Generation (RAG, for short). Most popular GenAI chatbots have general knowledge given information on the Internet and other resources. RAG systems create a database or repository of documents to be processed and queried against which is useful when the GenAI chatbot is used for a specific purpose, such as a university course.
RAG stands for Retrieval-Augmented Generation, a technique used in GenAI systems to improve the relevance, accuracy, and specificity of responses.
RAG enhances a generative AI model by integrating external knowledge retrieval. Instead of relying solely on the model's internal "knowledge," which is static and based on pretraining, RAG allows the system to retrieve up-to-date and domain-specific information from external data sources in real time.
There are two RAG mechanisms that UMD Virtual Agent provides.
The first mechanism is a Google Drive that comes with UMD Virtual Agent. This makes it easy to put relevant documents in Google Drive. UMD Virtual Agent will scan these documents and use the information to respond to prompts. UMD Virtual Agent preferentially uses this information over general information that most GenAI chatbots use. For example, let's say you teach a business course, BMGT355, a course on the foundations of marketing for non-majors.
You can put course documents into the Google Drive. The UMD Virtual Agent chatbot uses the BMGT355-related documents to answer student prompts. If students had asked a non-RAG GenAI chatbot the same question, they would get a general answer that's not specific to BMGT355. RAGs allow for targeted information resources that non GenAI chatbots don't.
In addition to getting information from a Google Drive, UMD Virtual Agent can also pull information from UMD websites. For example, UMD's University Health Center (UHC) is located at health.umd.edu. The UHC website has many webpages that provide detailed information about UHC services as well as answers to frequently asked questions (FAQ). There is a substantial amount of information that might require a skilled visitor to find what they want. If they can't easily locate the information, they may call UHC. This requires human interaction which would be limited to business hours.
Instead, the Health Center can point UMD Virtual Agent to health.umd.edu. UMD Virtual Agent will scan the main URL plus any UMD webpages under health.umd.edu. This revolutionizes the user experience. Rather than scour the website to find key information, a visitor can interact with a UMD Virtual Agent, asking it for information. UMD Virtual Agent provides a short response as well as links to the appropriate webpages where a more detailed answer can be found. This reduces the time for visitors to find information and greatly enhances their positive experience at the UHC website.
UMD Virtual Agent can be used on any UMD website to provide GenAI chatbot capabilities.
UMD Virtual Agent is a currently an umbrella over two products: Digital Student Assistant and Departmental Assistant. These two products share the same architecture underneath with some minor configuration differences. The two products are aimed at different target audiences despite common implementations. We say that Digital Study Assistant and Departmental Assistant are two types of UMD Virtual Agents.
Digital Study Assistant is target at UMD courses that want to offer students a GenAI chatbot that can answer course-related content. Instructors who use Digital Study Assistant are provided with a Google Drive where they can put course documents such as lecture notes, readings, sample quizzes, and transcripts to Panopto videos. Digital Study Assistant specifically uses course documents to generate responses to student inquiries.
Departmental Assistant, by contrast, is aimed at making it easier to find information at a UMD website. Departmental Assistant was used for the UMD's University Health Center GenAI chatbot mentioned in an earlier FAQ.
No. Both can use Google Drives and specify websites.
UMD Virtual Agent was developed by the AI Solutions Team. It's offered as two products: Digital Study Assistant for use in UMD courses, and Departmental Assistant for UMD websites.
The AI Solutions Team is part of the Division of Information Technology. In addition to UMD Virtual Agent, the team supports other AI solutions such as TerpAI. However, it does not support all AI services on campus, as many different campus groups have created a wealth of AI resources for the UMD campus.
Visit https://ai.umd.edu/resources/services to find a list of AI services provided by the Division of Information Technology.
If you have questions for the team, you can email dit-ais@umd.edu which will open a Service Now ticket.
Watch the following video which explains the process for Digital Study Assistant. The procedure is similar for Departmental Assistant.
You can either go to: http://go.umd.edu/DigitalStudyAssistant or http://go.umd.edu/DepartmentalAssistant. They both go to the same URL where you can request a UMD Virtual Agent.
Upon successful completion, you will receive an email with information about your UMD Virtual Agent and the next steps.
Yes, there is a monthly fee depending on the UMD Virtual Agent. Please contact the AI Solutions team for more information.
Owners of a UMD Virtual Agent manage their chatbot through an Admin Console. This can be found at the following URL: admin.chatbot.umd.edu. It will list all course chatbots where you are the owner.
You should have set up UMD Virtual Agent which requires a valid UMD email. If you're the primary owner of your UMD Virtual Agent, you will be a Chatbot Admin which permits you to access the Admin Console. CAS authentication is also required to access the Admin Console.
You can email the AI Solutions team at dit-ais@umd.edu. This will generate a Service Now ticket and a team member will respond to your questions.
Once you have requested either a Digital Study Assistant or a Departmental Assistant and been approved by the AI Solutions Team, you will get an email instructing you further on how to set up your UMD Virtual Agent.
You need to complete the prerequisites. The first step is to upload your syllabus which should either be a PDF file or a Word document. Next, you can add course material to your Google Drive. Digital Study Assistant will process the documents and syllabus which it will use to respond to student questions and prompts. Finally, you should go over the course FAQ, fixing any incorrect or incomplete answers and adding in answers where there are none. The link to the prerequisites are sent to you in an email once your Digital Study Assistant request has been approved.
This is the URL to a video explaining the process:
Your UMD Virtual Agent has a Google Drive where you can copy many file types, including Word documents, Google Doc, pdf, xls, Google Sheet, PPT, Google Slides, text, and text transcripts. You can also specify website URLs to have their HTML data scraped to answer questions. Finally, you can enter extra data directly into the Scripted Questions section of the Admin Console.
Yes, however, only PDFs are supported at this time. Also, since websites are recursively scraped, you need to review the list of PDFs on the Ingested Data section of the Admin Console. Click on the Web PDFs tab and decide which PDFs to scrape and which to exclude. Making changes to the included/excluded PDFs will trigger the web scrape process and will be completed along with the next file scan cycle instead of waiting until the next day's regularly scheduled web scrape.
The Admin Console provides a link to a stand-alone question window. This can be used to test your UMD Virtual Agent and share the link with potential users. A popup chatbot window can also be embedded in a web page if you have a web page administrator who can integrate it. A popup is often embedded in a UMD department website for those using Departmental Assistant. UMD Virtual Agent can not be integrated directly into Canvas.
It should only take a few seconds. If it takes more than a few seconds, there may be network interference or another issue. Email the AI Solutions team at dit-ais@umd.edu if your UMD Virtual Agent is not responding quickly enough.
Yes, sources are quoted in line with the answer using clickable links that open the sources box below the answer. There are also links to the material used to answer the question. This assumes the sources is placed in the Public folder.
Yes. If you don’t want UMD Virtual Agent to create citations/links to documents when answering questions, you can put those documents in the Private folder of your UMD Virtual Agent's Google Drive.
Yes, clicking on the thumbs-up will mark the question so it shows up in the Thumbs Up tab when you click Question review in the Admin Console sidebar menu. Clicking thumbs down will first ask for a brief description of the problem, then mark the question so it shows up in the Thumbs Down tab.
When you log into the Admin Console, the first option you see in the sidebar menu is Question Review. This has all the questions asked in the current month. Questions can be marked as reviewed, request developer review with comments you enter, or inspected. Finally, the data the UMD Virtual Agent used to answer the question can be inspected here.
Yes. The UMD Virtual Agent is configured to consider the last N questions when working on the current question. This, however, depends on what N is set to and whether there is enough space to hold all the current and previous data when building the complete set of information going to the LLM to generate an answer. Contact the AI Solutions team to discuss the history, performance, or settings involved here.
The Admin Console has a Question Review link on the sidebar menu. When you click it, you will see icon shaped like a piece of paper. It pops up a window that gives access to all of the source text chunks, chunks from previous questions, the prompt, and previous questions and answers. While the LLM can craft its answer drawing from any of that information, the one or two it says it used are highlighted.
As the UMD Virtual Agent admin, you can set the greeting and disclaimer by going to the Admin Console and clicking on “User Config” in the sidebar menu. If you want to change the name of your UMD Virtual Agent, please contact the AI Solutions team.
Yes, contact the AI Solutions team with a request to enable quiz mode. This feature will allow users to ask for a multiple-choice quiz generated from the data you provided. The AI Solutions team may need to make other updates to the configuration of your UMD Virtual Agent to enable the best quiz experience.
Mostly. If you create a directory in your Digital Study Assistant Google Drive called Assessments and put all the test material under it, Digital Study Assistant will offer resources about user questions that closely match assessment questions instead of responding with the actual answer. As an admin, you can set the text in the response when this feature is triggered. An assessment could be a quiz or homework questions. To prevent students from cheating, place quizzes and homeworks into the Assessments folder. Digital Study Assistant will avoid answering questions from these assessments.
Yes. However, this feature is only available for Digital Study Assistant. It is not available for Departmental Assistant.
Go to your Admin Console for your Digital Study Assistant and click on Virtual Agent Config in the sidebar menu. As an admin, you can use the Admin Console to toggle the Digital Study Assistant’s Test Mode. You can also set the text in the response when this Quiet Mode feature is used. The text underneath “Set Quiet Mode Response” and the other two “Set”s are editable fields. Click on them to change the text and hit enter if you want the change. Don't forget to turn Digital Study Assistant back on.
Yes, UMD Virtual Agent will mask things like social security numbers and other PII at the source, so the data is not stored in the system.
Yes, the LLM used to generate the answer to the question is configured to detect inappropriate and dangerous questions and provide a declined or supporting response.
Yes. In the Admin Console’s sidebar menu, click on User Analytics. The resulting page has a tab that graphs questions asked vs. time. There are also tabs for thumbs up and down questions asked.
Yes.In the Admin Console sidebar menu, click on Virtual Agent Analytics. There is an Intent Occurrence tab that graphs the number of times each chunk is used to answer a question. Each chunk has an intent name, just a few words describing what the goal of the user when asking a specific question, e.g. appointments could be the intent of “How do I make an appointment at the Health Center?”. This can help you understand the data that users are looking for the most so it can be refined as needed.
As an admin, you should review the questions, looking for incorrect answers and thumbs-down responses. Go to the Admin Console and select Question Review near the top of the sidebar menu. You can mark questions as reviewed to make it easier to keep track of progress.
When you find questions answered wrong, it's frequently because your UMD Virtual Agent ingested missing, conflicting, bad, or too much data.
Data is missing when UMD Virtual Agent’s response indicates it does not have information in its context to answer the question. Add files, web pages, or scripted questions to provide more data UMD Virtual Agent can choose from.
UMD Virtual Agent may be forced to choose from conflicting data and pick the wrong source. In this case, go to the Admin Console and select Question Review in the sidebar menu. On the right of the checkbox that appears left of the question, you see four icons. When you hover over the fourth icon, it displays a tooltip that says "See query sources". Click on that icon for the conflicting question. Find the source data chunk(s) that are not preferred and remove or fix them from the file or web page. If it's an unwanted web page, click on User Config in the sidebar menu of the Admin Console, and add it to the URLs to not Scrape list.
UMD Virtual Agent works best when clear semantic matches exist to the questions. If too many chunks of data match, the one you want may not match. In this case, you can reduce redundant data or increase the number of chunks returned to answer questions. Contact the AI Solutions team if you want to adjust settings like this.
Currently, your UMD Virtual Agent’s Google Drive is scanned for changes about every 20 minutes, and website data is scraped once a day, usually early in the morning.
Click on Ingested Data on the sidebar menu of the Admin Console. You will see the data ingested into your UMD Virtual Agent as multiple chunks for each file. The files are chunked to make it easier for the UMD Virtual Agent to match questions to the most relevant data it knows from your files.
UMD Virtual Agent can only work with the data it is provided. A list of broken web pages can be found by clicking on Ingested Data in the sidebar menu of the Admin Console. You can use it to find out what is out of date. Also, you can add URLs that are not included during the web scrape process. These could be broken URLs or just old/archived data you don't want UMD Virtual Agent to reference.
Yes, if you are making website changes and want them ingested right away and you have PDFs on your website, toggle on or off a small PDF. This will trigger the web scrape process for your UMD Virtual Agent, which will be completed along with the next Google Drive cycle.
At this point, using your own Google Drive instead of the one supplied with UMD Virtual Agent is problematic. We strongly suggest using the Google Drive provided by UMD Virtual Agent. Contact the AI Solutions Team if you have further questions.
To disable a document means to hide it from being scanned by UMD Virtual Agent.
To do so, if you haven't already, create a folder at the same level as Public and Private. Name this folder Ignore. Make sure to capitalize the "I" in Ignore. When you do this, you should see three folders: Public, Private, and Ignore.
Put the document in the Ignore folder. UMD Virtual Agent will not process any documents in that folder. It processes everything outside of the Ignore folder, which includes documents in the Public and Private folder.
UMD Virtual Agent uses a custom Retrieval Augmented Generation (RAG) technique that combines similarity-based data retrieval with generative-based artificial intelligence (GenAI) models.
No. UMD Virtual Agent is designed to answer questions only based on the data you provide. While it uses a powerful AI model to formulate answers from that data, it is instructed not to make up answers if it can not confidently create an answer based on the chunks of your ingested data it found related to the question. This direction can be modified. Contact the AI Solutions team if you want to change your UMD Virtual Agent's behavior prompt.
UMD Virtual Agent looks for chunks of your data that closely match the question. These chunks work best when they are concise and hold information about one topic. Chunks come from paragraphs of document files, rows of spreadsheets, short periods of transcribed audio, portions of PDFs, sections of web pages, HTML, and scripted questions.
Documents other than PDFs are usually broken into chunks by looking for paragraphs. Paragraphs separated by newlines help UMD Virtual Agent break up the document into better chunks. This can be verified by looking at the chunks by clicking on Ingested Data in the sidebar menu of the Admin Console. The text of the chunk is shown in the right column. Only the first 200 characters are displayed in the table cell, so it's best to verify chunks by looking at their beginning information.
PDFs are chunked by looking for text breaks and font size changes. Making these gaps larger may help UMD Virtual Agent find natural breaks in information. Multiple columns on a page are ok. If a topic starts at the bottom of one page and continues to the top of another page, it will be broken into different chunks. Tables and grids of information are hard for UMD Virtual Agent to turn into textual chunks of data. If some critical data is in a table, it may need to be entered into UMD Virtual Agent separately. Click on the Scripted Questions in the sidebar menu of the Admin Console to do so.
Keep each entry focused on one topic. In the Question cell, enter multiple versions of the question separated by newlines. Keep the total number of words in the answer under about 1000. Any links in the text should have https:// at the beginning.
UMD Virtual Agent can be configured with various AI models and other settings. It is best to contact the AI Solutions team with your concerns. They can adjust the model, number of chunks used, amount of data passed to the model, and other settings to help UMD Virtual Agent find and use the best data.
UMD Virtual Agent only uses semantic search to find the chunks that best match the words in the question. If too many chunks are needed to answer the question or the question has multiple parts, UMD Virtual Agent cannot find and use all the information necessary to answer the question.
Currently, UMD Virtual Agent does not have alternatives to documents or webpage. You can contact the AI Solutions team at dit-ais@umd.edu to discuss your specific situation and whether they can be accommodated by UMD Virtual Agent.