UMD Virtual Agent is an offering by the DIT AI Solutions team that provides GenAI chatbots using Retrieval-Augmented Generation (RAG, for short). Most popular GenAI chatbots have general knowledge given information on the Internet and other resources. RAG systems create a database or repository of documents to be processed and queried against which is useful when the GenAI chatbot is used for a specific purpose, such as a university course.
RAG stands for Retrieval-Augmented Generation, a technique used in GenAI systems to improve the relevance, accuracy, and specificity of responses.
RAG enhances a generative AI model by integrating external knowledge retrieval. Instead of relying solely on the model's internal "knowledge," which is static and based on pretraining, RAG allows the system to retrieve up-to-date and domain-specific information from external data sources in real time.
There are three RAG mechanisms that UMD Virtual Agent provides.
The first mechanism is a Google Drive that comes with UMD Virtual Agent. This makes it easy to put relevant documents in Google Drive. UMD Virtual Agent will scan these documents and use the information to respond to prompts. UMD Virtual Agent preferentially uses this information over general information that most GenAI chatbots use. For example, let's say you teach a business course, BMGT355, a course on the foundations of marketing for non-majors.
You can put course documents into the Google Drive. The UMD Virtual Agent chatbot uses the BMGT355-related documents to answer student prompts. If students had asked a non-RAG GenAI chatbot the same question, they would get a general answer that's not specific to BMGT355. RAGs allow for targeted information resources that non GenAI chatbots don't.
In addition to getting information from a Google Drive, UMD Virtual Agent can also pull information from UMD websites. For example, UMD's University Health Center (UHC) is located at health.umd.edu. The UHC website has many webpages that provide detailed information about UHC services as well as answers to frequently asked questions (FAQ). There is a substantial amount of information that might require a skilled visitor to find what they want. If they can't easily locate the information, they may call UHC. This requires human interaction which would be limited to business hours.
Instead, the Health Center can point UMD Virtual Agent to health.umd.edu. UMD Virtual Agent will scan the main URL plus any UMD webpages under health.umd.edu. This revolutionizes the user experience. Rather than scour the website to find key information, a visitor can interact with a UMD Virtual Agent, asking it for information. UMD Virtual Agent provides a short response as well as links to the appropriate webpages where a more detailed answer can be found. This reduces the time for visitors to find information and greatly enhances their positive experience at the UHC website.
UMD Virtual Agent can be used on any UMD website to provide GenAI chatbot capabilities.
The last option for a RAG mechanism is Service Now. A base URL as well as the Knowledge Base Categories must be provided and then any articles meeting those criteria will be used to answer user queries.
UMD Virtual Agent is a currently an umbrella over two products: Virtual Student Assistant and Departmental Assistant. These two products share the same architecture underneath with some minor configuration differences. The two products are aimed at different target audiences despite common implementations. We say that Virtual Study Assistant and Departmental Assistant are two types of UMD Virtual Agents. At times, we may refer to either as UMD Virtual Agents rather than their specific names.
Virtual Study Assistant targets UMD courses that want to offer students a GenAI chatbot that can answer course-related content. Instructors who use Virtual Study Assistant are provided with a Google Drive where they can put course documents such as lecture notes, readings, sample quizzes, and transcripts to Panopto videos. Virtual Study Assistant specifically uses course documents to generate responses to student inquiries.
Departmental Assistant, by contrast, is aimed at making it easier to find information at a UMD website. Departmental Assistant was used for the UMD's University Health Center GenAI chatbot mentioned in an earlier FAQ.
No. Both can use Google Drives, Service Now and web-scraping.
UMD Virtual Agent was developed by the DIT AI Solutions Team. It's offered as two products: Virtual Study Assistant for use in UMD courses, and Departmental Assistant for UMD websites.
The DIT AI Solutions Team is part of the Division of Information Technology. In addition to UMD Virtual Agent, the team supports other DIT AI Solutions such as TerpAI. However, it does not support all AI services on campus, as many different campus groups have created a wealth of AI resources for the UMD campus.
Visit https://ai.umd.edu/resources/services to find a list of AI services provided by the Division of Information Technology.
If you have questions for the team, you can email dit-ais@umd.edu which will open a Service Now ticket.
Watch the following video which explains the process for Virtual Study Assistant. The procedure is similar for Departmental Assistant.
You can request a UMD Virtual Agent here: UMD Virtual Agent Provisioning Request.
Upon successful completion, you will receive an email with information about your UMD Virtual Agent and next steps.
Note: If you use this method, you can skip the previous step.
Yes. To set up the Virtual Study Assistant in Canvas, scroll to the very bottom of the left sidebar menu and click the "UMD Virtual Agent" link.
Review the introductory information, follow the instructions, and click "Create Virtual Agent". If you’ve already created one, the system will notify you.
For step-by-step guidance on setting up Virtual Study Assistant using Canvas, additional documentation is available here: Virtual Study Assistant: Faculty Guide
There is currently no fee associated with the UMD Virtual Agents as they go through a trial period. However, in the future there may be monthly fee depending on the UMD Virtual Agent. Please contact the DIT AI Solutions team for more information.
Owners of a UMD Virtual Agent manage their chatbot through an Admin Console. This can be found at the following URL: admin.chatbot.umd.edu. It will list all course chatbots where you are the owner.
You should have set up UMD Virtual Agent which requires a valid UMD email. If you're the primary owner of your UMD Virtual Agent, you will be a Chatbot Admin which permits you to access the Admin Console. CAS authentication is also required to access the Admin Console.
Introductory material can be found here:
An intermediate guide can be found here:
You can email the DIT AI Solutions team at dit-ais@umd.edu. This will generate a Service Now ticket and a team member will respond to your questions.
Once you have requested either a Virtual Study Assistant or a Departmental Assistant and been approved by the DIT AI Solutions Team, you will get an email instructing you further on how to set up your UMD Virtual Agent.
You need to complete the prerequisites. The first step is to upload your syllabus which should either be a PDF file or a Word document. Next, you can add course material to your Google Drive. Virtual Study Assistant will process the documents and syllabus which it will use to respond to student questions and prompts. Finally, you should go over the course FAQ, fixing any incorrect or incomplete answers and adding in answers where there are none. The link to the prerequisites are sent to you in an email once your Virtual Study Assistant request has been approved.
This process is outlined in the "Request Process Overview" video above.
Your UMD Virtual Agent has a Google Drive where you can copy many file types, including Word documents, Google Doc, pdf, xls, Google Sheet, PPT, Google Slides, text, and text audio transcripts. You can also specify website URLs to have their HTML data scraped to answer questions. Finally, you can enter extra data directly into the Scripted Questions section of the Admin Console.
Yes, however, only PDFs are supported at this time. PDFs from websites are excluded by default. If you'd like to include web PDFs, you'll need to review the list of PDFs on the Ingested Sources section of the Admin Console. In the "Ingested Sources" tab, web PDFs will be listed as type 'PDFPAGE'. You can use the toggle on the right under 'Included' to either include or exclude selected PDFs. Marking a currently excluded PDF as included will cause an immediate ingest of that PDF. However, if you mark a currently included PDF as excluded the change will be reflected after the next nightly webscraping event.
The Admin Console provides a link to a stand-alone question window. This can be used to test your UMD Virtual Agent and share the link with potential users. A popup chatbot window can also be embedded in a web page if you have a web page administrator who can integrate it. A popup is often embedded in a UMD department website for those using Departmental Assistant. UMD Virtual Agent can not be integrated directly into Canvas.
It should only take a few seconds. If it takes more than a few seconds, there may be network interference or another issue. Email the DIT AI Solutions team at dit-ais@umd.edu if your UMD Virtual Agent is not responding quickly enough.
Yes, sources are quoted in line with the answer using clickable links that open the sources box below the answer. There are also links to the material used to answer the question. This assumes the sources are placed in the Public folder. Sources placed within the Private folder will be referenced as 'Private Source'.
Yes. If you don’t want UMD Virtual Agent to create citations/links to documents when answering questions, you can put those documents in the Private folder of your UMD Virtual Agent's Google Drive.
Yes, clicking on the thumbs-up will mark the question so it shows up in the Thumbs Up tab when you click Question review in the Admin Console sidebar menu. Clicking thumbs down will first ask for a brief description of the problem, then mark the question so it shows up in the Thumbs Down tab of the Question Review page in the Admin Console.
When you log into the Admin Console, the first option you see in the sidebar menu is Question Review. This has all the questions asked in the current month. For Virtual Agents that experience a high volume of questions in a month, questions may be displayed by day instead. Questions can be marked as reviewed, test, or request developer review which will prompt you to provide comments on why the review is necessary. Finally, the data the UMD Virtual Agent used to answer the question can be inspected here.
Yes. The UMD Virtual Agent is configured to consider the last N questions when working on the current question. This, however, depends on what N is set to and whether there is enough space to hold all the current and previous data when building the complete set of information going to the LLM to generate an answer. Contact the DIT AI Solutions team to discuss the history, performance, or settings involved here.
The Admin Console has a Question Review link on the sidebar menu. When you click it, you will see icon shaped like a piece of paper. It pops up a window that gives access to all of the source text chunks, chunks from previous questions, the prompt, and previous questions and answers. While the LLM can craft its answer drawing from any of that information, the one or two it says it used are highlighted.
As the UMD Virtual Agent admin, you can set the greeting and disclaimer by going to the Admin Console and clicking on “User Config” in the sidebar menu. If you want to change the name of your UMD Virtual Agent, please contact the DIT AI Solutions team.
Yes, contact the DIT AI Solutions team with a request to enable quiz mode. This feature will allow users to ask for a multiple-choice quiz generated from the data you provided. The DIT AI Solutions team may need to make other updates to the configuration of your UMD Virtual Agent to enable the best quiz experience.
Mostly. If you create a directory in your Virtual Study Assistant Google Drive called Assessments and put all the test material under it, Virtual Study Assistant will offer resources about user questions that closely match assessment questions instead of responding with the actual answer. As an admin, you can set the text in the response when this feature is triggered. An assessment could be a quiz or homework questions. To prevent students from cheating, place quizzes and homeworks into the Assessments folder. Virtual Study Assistant will avoid answering questions from these assessments.
Yes. However, this feature is only available for Virtual Study Assistant. It is not available for Departmental Assistant.
Go to your Admin Console for your Virtual Study Assistant and click on Virtual Agent Config in the sidebar menu. As an admin, you can use the Admin Console to toggle the Virtual Study Assistant’s Test Mode. You can also set the text in the response when this Quiet Mode feature is used. The text underneath “Set Quiet Mode Response” and the other two “Set”s are editable fields. Click on them to change the text and hit enter if you want the change. Don't forget to turn Virtual Study Assistant back on. You can also set quiet periods by using the 'Outage Start' and 'Outage End' variables under the 'General' Section on the Virtual Agent Config page. This will allow you to set a planned outage in EST without having to visit the Admin Console directly before and after the outage to engage Quiet Mode.
Yes, UMD Virtual Agent will mask things like social security numbers and other PII at the source, so the data is not stored in the system.
Yes, the LLM used to generate the answer to the question is configured to detect inappropriate and dangerous questions and provide a declined or supporting response.
Yes. In the Admin Console’s sidebar menu, click on User Analytics. The resulting page has a tab that graphs questions asked vs. time. There are also tabs for thumbs up and down questions asked.
Yes. In the Admin Console sidebar menu, click on Virtual Agent Analytics. There is an Intent Occurrence tab that graphs the number of times each chunk is used to answer a question. Each chunk has an intent name, just a few words describing what the goal of the user when asking a specific question, e.g. appointments could be the intent of “How do I make an appointment at the Health Center?”. This can help you understand the data that users are looking for the most so it can be refined as needed.
The sample questions will only be created or adjusted when the syllabus is replaced in the Admin Console. We are first checking to see if any topics have changed. If the topics in the syllabus have not changed then new sample questions will not be created. However, a user can go into the ‘Sample Questions’ page on the Admin Console and either alter, delete or add existing sample questions.
The answers to the questions are based on the course content. If AI cannot generate an answer based on the existing course content then it will label the answer as ‘Course Content Not Available’. If the user believes that they’ve added content to answer the question, they can press the button in the upper right to ‘Re-run sample question validation’. This will only update answers for questions that have ‘Course Content Not Available’, questions with an existing answer will not be impacted.
Validation is not automatic after the initial generation of the questions. The user must manually press the button to run this validation process, however, questions that were not able to be answered will not appear in the dropdown for students to select in the chatbot UI. However, if a new syllabus is uploaded and new sample questions are created then those questions will automatically run through the validation process.
UMD Virtual Agents are automatically configured to have user questions stored in a database and presented on the Admin Console for administrators to review. However, if you have sensitive or private information, there is an option to forego saving question data. In this case, no question data will be shown in the Admin Console and administrators will not have the ability to review questions.
There are two options for CAS integration with the UMD Virtual Agents.
UMD Virtual Agents detects numbers related to university id’s and social security numbers and masks the data once the message is sent as well as when the data is stored and shown on the Admin Console.
As an admin, you should review the questions, looking for incorrect answers and thumbs-down responses. Go to the Admin Console and select Question Review near the top of the sidebar menu. You can mark questions as reviewed to make it easier to keep track of progress.
When you find questions answered wrong, it's frequently because your UMD Virtual Agent ingested missing, conflicting, bad, or too much data.
Data is missing when UMD Virtual Agent’s response indicates it does not have information in its context to answer the question. Add files, web pages, or scripted questions to provide more data UMD Virtual Agent can choose from.
UMD Virtual Agent may be forced to choose from conflicting data and pick the wrong source. In this case, go to the Admin Console and select Question Review in the sidebar menu. On the right of the checkbox that appears left of the question, you see four icons. When you hover over the fourth icon, it displays a tooltip that says "See query sources". Click on that icon for the conflicting question. Find the source data chunk(s) that are not preferred and remove or fix them from the file or web page. If it's an unwanted web page, click on Ingested Sources in the sidebar menu of the Admin Console, and use the toggle to exclude that webpage from the included data. The webpage will be excluded immediately, however if you want to revert it to be included again this change will be reflected after the nightly webscraping event.
UMD Virtual Agent works best when clear semantic matches exist to the questions. If too many chunks of data match, the one you want may not match. In this case, you can reduce redundant data or increase the number of chunks returned to answer questions. Contact the DIT AI Solutions team if you want to adjust settings like this.
Currently, your UMD Virtual Agent’s Google Drive is scanned for changes about every couple of minutes, and website data is scraped once a day, usually early in the morning or overnight.
Click on Ingested Sources on the sidebar menu of the Admin Console. You will see the data ingested into your UMD Virtual Agent as multiple chunks for each file. The files are chunked to make it easier for the UMD Virtual Agent to match questions to the most relevant data it knows from your files.
UMD Virtual Agent can only work with the data it is provided. A list of broken web pages can be found by clicking on Ingested Sources in the sidebar menu of the Admin Console. You can use it to find out what is out of date. These could be broken URLs or just old/archived data you don't want UMD Virtual Agent to reference.
If you need changes reflected immediately, please email the DIT AI Solutions team at dit-ais@umd.edu. Otherwise, the changes will be reflected after the next nightly webscraping event.
At this point, using your own Google Drive instead of the one supplied with UMD Virtual Agent is problematic. We strongly suggest using the Google Drive provided by UMD Virtual Agent. Contact the DIT AI Solutions Team if you have further questions.
To disable a document means to hide it from being scanned by UMD Virtual Agent.
To do so, if you haven't already, create a folder at the same level as Public and Private. Name this folder Ignore. Make sure to capitalize the "I" in Ignore. When you do this, you should see three folders: Public, Private, and Ignore.
Put the document in the Ignore folder. UMD Virtual Agent will not process any documents in that folder. It processes everything outside of the Ignore folder, which includes documents in the Public and Private folder.
UMD Virtual Agent uses a custom Retrieval Augmented Generation (RAG) technique that combines similarity-based data retrieval with generative-based artificial intelligence (GenAI) models.
No. UMD Virtual Agent is designed to answer questions only based on the data you provide. While it uses a powerful AI model to formulate answers from that data, it is instructed not to make up answers if it can not confidently create an answer based on the chunks of your ingested data it found related to the question. This direction can be modified. Contact the DIT AI Solutions team if you want to change your UMD Virtual Agent's behavior prompt.
UMD Virtual Agent looks for chunks of your data that closely match the question. These chunks work best when they are concise and hold information about one topic. Chunks come from paragraphs of document files, rows of spreadsheets, short periods of transcribed audio, portions of PDFs, sections of web pages, HTML, and scripted questions.
Documents other than PDFs are usually broken into chunks by looking for paragraphs. Paragraphs separated by newlines help UMD Virtual Agent break up the document into better chunks. This can be verified by looking at the chunks by clicking on Ingested Sources in the sidebar menu of the Admin Console. Only the title of the item is displayed here, if you'd like to see a preview of the text in the chunk you can select the 'Ingest Details' tab. The text of the chunk is shown in the right column. Only the first 200 characters are displayed in the table cell, so it's best to verify chunks by looking at their beginning information.
PDFs are chunked by looking for text breaks and font size changes. Making these gaps larger may help UMD Virtual Agent find natural breaks in information. Multiple columns on a page are ok. If a topic starts at the bottom of one page and continues to the top of another page, it will be broken into different chunks. Tables and grids of information are hard for UMD Virtual Agent to turn into textual chunks of data. If some critical data is in a table, it may need to be entered into UMD Virtual Agent separately. Click on the Scripted Questions in the sidebar menu of the Admin Console to do so.
Keep each entry focused on one topic. In the Question cell, enter multiple versions of the question separated by newlines. Keep the total number of words in the answer under about 1000. Any links in the text should have https:// at the beginning.
UMD Virtual Agent can be configured with various AI models and other settings. It is best to contact the DIT AI Solutions team with your concerns. They can adjust the model, number of chunks used, amount of data passed to the model, and other settings to help UMD Virtual Agent find and use the best data.
UMD Virtual Agent uses semantic search to find the chunks that best match the words in the question. If too many chunks are needed to answer the question or the question has multiple parts, UMD Virtual Agent cannot find and use all the information necessary to answer the question. In special cases, the DIT AI Solutions can enable keyword search which allows the Virtual Agent to search by non-dictionary words. This is useful in cases where users may be searching for contract numbers or class IDs. Please contact the DIT AI Solutions team if you'd like to enable keyword search for you Virtual Agent.
Currently, UMD Virtual Agent does not have alternatives to documents, Service Now or webpage. You can contact the DIT AI Solutions team at dit-ais@umd.edu to discuss your specific situation and whether they can be accommodated by UMD Virtual Agent.