UMD Virtual Agents

Generative Artificial Intelligence (GenAI)- powered assistants are software programs that use AI and natural language processing to interact with users in a conversational manner. They are designed to simulate conversational interactions via text, voice, or both. This article covers essential aspects, including capabilities, benefits, limitations, features, etc.

The UMD Virtual Agent is an AI-powered assistant designed to assist users with inquiries related to the University of Maryland's many departments, courses, and personnel. There are two main types of UMD Virtual Agents:  Departmental Assistant and Virtual Study Assistant.

Virtual Study Assistant

Virtual Study Assistants are for specific university academic courses. Only a controlled list of users, such as students, professors, and TAs, can access this UMD Virtual Agent chatbot. Extra features like quiz mode, Panopto transcript/video indexing, and the ability to disable the UMD Virtual Agent during tests are available. 

Departmental Assistant

Departmental Assistants can be created for a department or a smaller unit within a department. It can be internal or public-facing.  Generally, these assistants provide information against a finite set of departmental documentation. 

Top

Key features of UMD Virtual Agents

Virtual Agents

UMD Virtual Agents can be easily integrated into the web platforms and accessed with a click of a button.

Google Drive

Your UMD Virtual Agent comes with a Google Drive (to deposit the files for the virtual agent to ingest). Accepted file formats are Google Docs and text-based PDFs.

AI Admin Console

The AI Admin Console is a tool the administrative staff can use to review assistants activity, including the number of interactions, user engagement metrics, response accuracy rates, and peak usage times, as well as monitor assistants conversations and performance in real time.

Question review 

Use the siloed Question Review panel to monitor and administer the actions on the questions the assistants was asked. The questions are sorted by different categories.

• The All tab serves as a comprehensive repository, displaying every question posed to the assistants within the specified period selected via the left-side menu. This default view ensures a complete overview of the virtual agent's interactions.
• The Unknown tab in the virtual agent's interface categorizes questions that the virtual agent could not answer due to gaps in its knowledge base, queries outside its domain expertise, or the complexity or incoherence of the questions. These are typically flagged for review to enhance the virtual agent's performance.
• The Known tab filters the virtual agent's interactions to show only the questions it successfully recognized and addressed, representing the subset of All that excludes the Unknown queries.
• The Thumbs up tab categorizes the questions flagged as good or useful by the end users using the thumbs up button in the virtual agent's interface.
• The Thumbs down tab is another important silo of the questions that were marked as unsatisfactorily answered, signaling areas where the virtual agent's responses could be improved. The feedback provided by the end users could help further diagnose the answers.
• The Reviewed tab stores the questions that were marked as reviewed by the administrators from the other tabs. Use this icon button to mark a question as reviewed.
• The Needs Dev Eval & Dev Feedback tabs can be used to mark the questions needing the developer Review and the developer's Feedback. Use the "eye" icon to request a developer review on a question.
• The Sources icon will pop up a window that shows all of the material that was deemed to be relevant to the question. This source material is what is presented to the LLM so it can formulate the written answer to the question.
  • Each source has a title, intent and type. The type of source can be things like web, pdf, gdoc, gsheet and qna. The qna source means it is a scripted question entered and managed in this admin app. The title is the full name and or link to the source material. The intent is a short description of the source. This intent is also used in the analytics charts. This allows a user to match intents used heavily with the material they came from.
  • Previous sources are material that matched the previous questions in the conversation.
  • The prompt is the text used to direct the LLM's behavior. Generally, it tells the LLM what role it has and directs it to only use material provided in these sources.
  • Historical Conversation lists the previous questions and answers included in the conversation. This is limited to the last few questions.

Scripted Questions 

The Scripted Questions section is where predetermined questions and their corresponding scripted responses are stored, enabling the assistants to provide consistent answers to the frequently asked queries. 

User Analytics

The data visualizations in the User Analytics page serve as a comprehensive dashboard for administrative staff to monitor the UMD Virtual Agent's performance. It helps in identifying patterns in query volume, discerning which intents are most frequently misunderstood or unrecognized by the assistants, and evaluating user satisfaction over time. This information is crucial for pinpointing areas where the assistants may require further training or refinement to improve user experience and efficiency.

User Config 

The User Config page is an administrative tool where administrators can personalize the virtual agent's name, set up the initial greeting message, and manage the URLs the virtual agent is allowed to scrape for information separated by a new line for multiple entries. Here, they can also specify which URLs should be excluded from the virtual agent's data scraping activities to prevent unauthorized ingestion of content. To facilitate ease of access to resources, the page also features a direct link to Google Drive.

Feedback

The feedback feature in the interface is a user engagement tool designed to gauge satisfaction with the virtual agent's responses.

• Additionally, it offers a text box where users can articulate specific issues they encountered.
• This feedback mechanism is integral to enhancing the virtual agent's performance by allowing developers to collect and analyze user experience data and make informed improvements.

Top

UMD Virtual Agent FAQ

What types of UMD Virtual Agents are there?

There are two types of UMD Virtual Agents: Departmental Assistant and Virtual Study Assistant.

What is a Departmental Assistant?

This can be created for a department or a smaller unit within a department. It can be internal or public-facing.

What is Virtual Study Assistant?

Virtual Study Assistants are for specific university academic courses.  Only a controlled list of users, such as students, professors, and TAs, can access this bot. Extra features like quiz mode, Panopto transcript/video indexing, and the ability to disable the virtual agent during tests are available. This feature is now integrated into ELMS-Canvas and Panopto.

How do I get a UMD Virtual Agent?

You can request a UMD Virtual Agent using this Service Now page. If you need assistance with this process, contact the DIT AI Solutions Team at dit-ais@umd.edu.

Is there a fee for UMD Virtual Agent?

No. For the time being, DIT is offering UMD Virtual Agents free of charge.

Where do I go to administer my UMD Virtual Agent?

The administrative features for your UMD Virtual Agent are located at: admin.chatbot.umd.edu. The administrative web application is called the Admin Console.

This is where you can find the link to your Google Drive, add web pages to scrape and control other settings for your virtual agent.

Can anyone access the Admin Console?

You need a UMD email, to be assigned as a chatbot admin, and to use CAS authentication to access the Admin Console.

How do I get help with UMD Virtual Agents?

The DIT AI Solutions Team can be reached by email at dit-ais@umd.edu.

Top

Key capabilities of UMD Virtual Agents

Natural Language Processing (NLP)

NLP allows UMD Virtual Agents to understand and generate human-like text. It processes and analyzes large amounts of natural language data to respond in a contextually relevant and coherent way. This capability enables the assistants to engage in conversations, answer questions, and perform language-based tasks with a high level of proficiency.

Retrieval-Augmented Generation (RAG)

RAG is a technique that combines the power of large language models like ChatGPT, Claude with external information retrieval. It enables the assistants to pull in information from a Domain knowledge of data particular to the departments the virtual agent is used in, to supplement its pre-trained knowledge. This is particularly useful for answering questions that require up-to-date or specialized information that is grounded in the curated KB.

Understanding complex queries

UMD Virtual Agents are equipped to understand and respond to complex queries. This means the agent can handle multi-part questions, infer the intent behind a query, and provide answers that consider all aspects of the question. This is achieved through sophisticated algorithms that analyze the structure and semantics of the query.

Context retention

This refers to UMD Virtual Agent's ability to remember and utilize the context of a conversation over up to two exchanges. It can recall earlier parts of the conversation and use this information to make responses more relevant and personalized. Context retention is crucial for maintaining a coherent and logical flow in conversations.

Feedback loop

The feedback the virtual agent receives through the thumbs-up/thumbs-down feature can help the admins further tune the content and merge the curated data with the KB, thus improving the UMD Virtual Agent's qualitative performance.

Faster development

A streamlined development framework allows for the faster development and deployment of UMD Virtual Agents.

Top

Key benefits of UMD Virtual Agents

24/7 availability

UMD Virtual Agents provide round-the-clock assistance, answering student, faculty and staff queries anytime, which is especially beneficial for universities with a large or international student body across different time zones.

Instant response

UMD Virtual Agents provide immediate answers to user queries, significantly reducing wait times compared to human-operated services. This also reduces the admin staff overload via email and call volume.

Handling high volume of interactions

During peak periods like admissions or exam seasons, UMD Virtual Agents can efficiently handle a high volume of student interactions, reducing the pressure on the staff.

Cost-effectiveness

UMD Virtual Agents help departments save on labor costs and resource allocation by automating responses to common queries.

Scalability

UMD Virtual Agents can easily scale up to handle an increasing number of interactions without needing additional significant resources, unlike human-operated services, which would require more staff.

Personalization

UMD Virtual Agents can retain context and learn from past interactions, providing more personalized and contextually relevant responses and enhancing the user experience.

Data Insights

They can gather and analyze user interaction data to provide valuable insights into customer behavior and preferences, helping businesses improve their services and strategies.

Top

Key challenges of UMD Virtual Agents

Understanding context and nuance

UMD Virtual Agents cannot understand the context and nuances of human conversation. They might misinterpret sarcasm, idioms, or complex sentences, leading to irrelevant responses.

Dependency on quality and diversity of training data

The performance of an UMD Virtual Agents heavily depends on the quality and diversity of the data it was trained on. Biases in the training data can lead to biased responses, which can be problematic, especially in sensitive topics.

Access and usage

• Eligibility: Faculty and staff may order the service through ServiceNow.
• How to Access: The assistant owner will receive an email with access instructions once provisioned.
• Usage Instructions: Several ServiceNow KB articles are available.
• UMD Virtual Agents
• UMD Virtual Agents FAQ
• UMD Virtual Agent Guide
• UMD Virtual Agent Admin Console Guide

Top

Planned maintenance

Environment	Maintenance Day	Frequency	Details
QA	Monday	Twice a month (Every other Monday)	Regular patching and updates
Production	Wednesday	Twice a month (Every other Wednesday)	Regular patching and updates

Notes

Quality Assurance (QA) Environment: Maintenance is conducted every other Monday to ensure that all updates and patches are tested before moving to production.

Production Environment: Maintenance is conducted every other Wednesday to ensure minimal disruption and to apply all necessary patches and updates verified in the QA environment.

This schedule ensures regular maintenance while minimizing downtime and ensuring system stability and security.

Top

Contact

The DIT AI Solutions team is continuously implementing new feature updates. We are happy to hear from UMD Virtual Agent users about features they would like to see added to UMD Virtual Agents!

For more information, please contact the us at dit-ais@umd.edu.

Top