UMD Virtual Agents

Artificial Intelligence (AI)- powered assistants are software programs that use AI and natural language processing to interact with users in a conversational manner. They are designed to simulate conversational interactions via text, voice, or both. This article covers essential aspects, including capabilities, benefits, limitations, features, etc.

The UMD Virtual Agent is an AI-powered assistant designed to assist users with inquiries related to the University of Maryland's many departments, courses, and personnel. There are four types of UMD AI assistants:  Departmental and Academic.

Digital Study Assistant

Academic Digital Study Assistants are for specific university academic courses. Only a controlled list of users, such as students, professors, and TAs, can access this bot. Extra features like quiz mode, Panopto transcript/video indexing, and the ability to disable the bot during tests are available. 

Departmental Assistant

Departmental assistants can be created for a department or a smaller unit within a department. It can be internal or public-facing.  Generally, these assistants provide information against a finite set of departmental documentation. 

Top

Key features of UMD Virtual Agents

Assistants

The assistants can be easily integrated into the web platforms and accessed with a click of a button.

Google Drive

The admin framework comes with a Google Drive (to deposit the files for the assistants to ingest. Accepted file formats are Google Docs and text-based PDFs.

AI Admin Console

The AI Admin Console is a tool the administrative staff can use to review assistants activity, including the number of interactions, user engagement metrics, response accuracy rates, and peak usage times, as well as monitor assistants conversations and performance in real time.

Question review 

Use the siloed Question Review panel to monitor and administer the actions on the questions the assistants was asked. The questions are sorted by different categories.

• The All tab serves as a comprehensive repository, displaying every question posed to the assistants within the specified period selected via the left-side menu. This default view ensures a complete overview of the assistants's interactions.
• The Unknown tab in the assistants's interface categorizes questions that the bot could not answer due to gaps in its knowledge base, queries outside its domain expertise, or the complexity or incoherence of the questions. These are typically flagged for review to enhance the assistants's performance.
• The Known tab filters the assistants's interactions to show only the questions it successfully recognized and addressed, representing the subset of All that excludes the Unknown queries.
• The Thumbs up tab categorizes the questions flagged as good or useful by the end users using the thumbs up button in the assistants’s interface.
• The Thumbs down tab is another important silo of the questions that were marked as unsatisfactorily answered, signaling areas where the assistants’s responses could be improved. The feedback provided by the end users could help further diagnose the answers.
• The Reviewed tab stores the questions that were marked as reviewed by the administrators from the other tabs. Use this icon button to mark a question as reviewed.
• The Needs Dev Eval & Dev Feedback tabs can be used to mark the questions needing the developer Review and the developer's Feedback. Use the "eye" icon to request a developer review on a question.
• The Sources icon will pop up a window that shows all of the material that was deemed to be relevant to the question. This source material is what is presented to the LLM so it can formulate the written answer to the question.
  • Each source has a title, intent and type. The type of source can be things like web, pdf, gdoc, gsheet and qna. The qna source means it is a scripted question entered and managed in this admin app. The title is the full name and or link to the source material. The intent is a short description of the source. This intent is also used in the analytics charts. This allows a user to match intents used heavily with the material they came from.
  • Previous sources are material that matched the previous questions in the conversation.
  • The prompt is the text used to direct the LLM's behavior. Generally, it tells the LLM what role it has and directs it to only use material provided in these sources.
  • Historical Conversation lists the previous questions and answers included in the conversation. This is limited to the last few questions.

Scripted Questions 

The Scripted Questions section is where predetermined questions and their corresponding scripted responses are stored, enabling the assistants to provide consistent answers to the frequently asked queries. 

User Analytics

The data visualizations in the User Analytics page serve as a comprehensive dashboard for administrative staff to monitor the AI assistants's performance. It helps in identifying patterns in query volume, discerning which intents are most frequently misunderstood or unrecognized by the assistants, and evaluating user satisfaction over time. This information is crucial for pinpointing areas where the assistants may require further training or refinement to improve user experience and efficiency.

User Config 

The User Config page is an administrative tool where administrators can personalize the assistants's name, set up the initial greeting message, and manage the URLs the bot is allowed to scrape for information separated by a new line for multiple entries. Here, they can also specify which URLs should be excluded from the bot's data scraping activities to prevent unauthorized ingestion of content. To facilitate ease of access to resources, the page also features a direct link to Google Drive.

Feedback

The feedback feature in the interface is a user engagement tool designed to gauge satisfaction with the assistants's responses.

• Additionally, it offers a text box where users can articulate specific issues they encountered.
• This feedback mechanism is integral to enhancing the assistants's performance by allowing developers to collect and analyze user experience data and make informed improvements.

Top

UMD Virtual Agent FAQ

What types of UMD AI assistants are there?

There are two types of UMD AI assistants:  Departmental and Academic.

What is a Departmental AI assistant?

This can be created for a department or a smaller unit within a department. It can be internal or public-facing.

What is an Academic AI assistant?

Academic assistants are for specific university academic courses.  Only a controlled list of users, such as students, professors, and TAs, can access this bot. Extra features like quiz mode, Panopto transcript/video indexing, and the ability to disable the bot during tests are available. 

How do I get an assistant?

You can request an assistant using this Service Now page 

Is there a fee for UMD assistants?

Yes, there is a monthly fee depending on the type of assistants. The fee is described on the Service Now page.

Where do I go to set up my assistants?

The  at  UMD AI assistants Admin Application web page is where you can find the link to your Google Drive, add web pages to scrape and control other settings for your bot.

Can anyone access that assistant Admin UI web page?

You need a UMD email, to be assigned as an assistant admin, and to use CAS authentication to access the assistant admin UI web page.

How do I contact the UMD AI assistants office?

We can be reached by email at ese-sws@umd.edu.

Top

Key capabilities of Generative AI assistants

Natural Language Processing (NLP)

NLP allows the assistants to understand and generate human-like text. It processes and analyzes large amounts of natural language data to respond in a contextually relevant and coherent way. This capability enables the assistants to engage in conversations, answer questions, and perform language-based tasks with a high level of proficiency.

Retrieval-Augmented Generation (RAG)

RAG is a technique that combines the power of large language models like ChatGPT, Claude with external information retrieval. It enables the assistants to pull in information from a Domain knowledge of data particular to the departments the bot is used in, to supplement its pre-trained knowledge. This is particularly useful for answering questions that require up-to-date or specialized information that is grounded in the curated KB.

Understanding complex queries

The AI assistants is equipped to understand and respond to complex queries. This means the bot can handle multi-part questions, infer the intent behind a query, and provide answers that consider all aspects of the question. This is achieved through sophisticated algorithms that analyze the structure and semantics of the query.

Context retention

This refers to the assistants's ability to remember and utilize the context of a conversation over up to two exchanges. It can recall earlier parts of the conversation and use this information to make responses more relevant and personalized. Context retention is crucial for maintaining a coherent and logical flow in conversations.

Feedback loop

The feedback the bot receives through the thumbs-up/thumbs-down feature can help the admins further tune the content and merge the curated data with the KB, thus improving the AI assistants's qualitative performance.

Faster development

A streamlined development framework allows for the faster development and deployment of AI assistants.

Top

Key benefits of Generative AI assistants

24/7 availability

AI assistants provide round-the-clock assistance, answering student, faculty and staff queries anytime, which is especially beneficial for universities with a large or international student body across different time zones.

Instant response

Assistants provide immediate answers to user queries, significantly reducing wait times compared to human-operated services. This also reduces the admin staff overload via email and call volume.

Handling high volume of interactions

During peak periods like admissions or exam seasons, assistants can efficiently handle a high volume of student interactions, reducing the pressure on the staff.

Cost-effectiveness

Assistants help departments save on labor costs and resource allocation by automating responses to common queries.

Scalability

Assistants can easily scale up to handle an increasing number of interactions without needing additional significant resources, unlike human-operated services, which would require more staff.

Personalization

GenAI assistants can retain context and learn from past interactions, providing more personalized and contextually relevant responses and enhancing the user experience.

Data Insights

They can gather and analyze user interaction data to provide valuable insights into customer behavior and preferences, helping businesses improve their services and strategies.

Top

Key challenges of Generative AI assistants

Understanding context and nuance

AI assistants cannot understand the context and nuances of human conversation. They might misinterpret sarcasm, idioms, or complex sentences, leading to irrelevant responses.

Dependency on quality and diversity of training data

The performance of an AI assistant heavily depends on the quality and diversity of the data it was trained on. Biases in the training data can lead to biased responses, which can be problematic, especially in sensitive topics.

Access and usage

• Eligibility: Faculty and staff may order the service through ServiceNow.
• How to Access: The assistant owner will receive an email with access instructions once provisioned.
• Usage Instructions: Several ServiceNow KB articles are available.
• UMD Virtual Agents
• UMD Virtual Agents FAQ
• UMD Virtual Agent Guide
• UMD Virtual Agent Admin Console Guide

Top

Planned maintenance

Environment	Maintenance Day	Frequency	Details
QA	Monday	Twice a month (Every other Monday)	Regular patching and updates
Production	Wednesday	Twice a month (Every other Wednesday)	Regular patching and updates

Notes

Quality Assurance (QA) Environment: Maintenance is conducted every other Monday to ensure that all updates and patches are tested before moving to production.

Production Environment: Maintenance is conducted every other Wednesday to ensure minimal disruption and to apply all necessary patches and updates verified in the QA environment.

This schedule ensures regular maintenance while minimizing downtime and ensuring system stability and security.

Top

Contact

The AIS team is continuously implementing new feature updates. We are happy to hear from assistant users about features they would like to see added to assistants!

For more information, please contact the Software Services AI team at ese-sws@umd.edu.

Top