Identifying User Requirements Using LLMs

Dustin O’Brien Spencer Presley Xiaohong Wang Shuangquan Wang

Department of Computer Science Department of Computer Science Department of Computer Science Department of Computer Science
Salisbury University Salisbury University Salisbury University Salisbury University
Salisbury, United States Salisbury, United States Salisbury, United States Salisbury, United States
[email protected] [email protected] [email protected] [email protected]

Abstract— Saltwater intrusion caused by rising sea levels, that there exists a great deal of practicality and benefits to their
droughts, and increased freshwater pumping in the coast and work. This research aims to achieve some of the same benefits
tidal regions poses a significant threat to people's daily lives and to this approach, but by using tools to improve performance of
industrial and agricultural production. Many organizations and existing LLMs, making it more affordable and more
individuals have come together to develop various products to accessible.
reduce its impact. A major challenge in this is that different
water users have different water usage types such as irrigation, The rest of this paper is organized as follows. Section II
power generation, sand & gravel washing, product introduces the technologies used in this project. Section III &
manufacturing, drinking water, etc. and it is essential to learn IV describe the backend and frontend systems of this
and understand the needs of each user group so that appropriate application respectively. Section V describes the user
services can be provided and used effectively. To effectively interview processing. Section VI discusses results and Section
identify the diverse requirements from the water users, in this VII concludes this paper.
paper, we proposed a method to use Large Language Model
(LLM) to analyze water user interview transcripts and construct II. WHY USE LLMS?
an AI-powered chat system. The system supports quick and
accurate understanding of client water usage requirements under A. What is an LLM?
the situation of saltwater intrusion.
LLMs are a recently developed type of AI that is focused
I. INTRODUCTION on the understanding and generation of human language. 3 In
Saltwater intrusion is an increasingly pressing climate other words, they are capable of taking an input text and
issue, characterized by the mixing of saltwater with freshwater generating an output text based on that input text. A common
due to rising sea levels1, droughts, and increased freshwater application of LLMs is chat-bots, which takes a user prompt
pumping.2 This phenomenon poses a significant threat to and creates their own text response. The most popular of these
people's daily lives and industrial and agricultural production. is OpenAI’s ChatGPT.3
These concerns have generated demand for products to B. How do LLMs fulfill our needs?
monitor, visualize, and predict water salt concentration.
However, due to the diverse and user-specific demands of the For our research, we need to extract requirements about
water usage clients, it is important to identify each client's water usage from interview text documents. An LLM
unique needs through user interviews, surveys, etc. To quickly architecture is well-suited for this task due to it’s ability to
and correctly synthesize the results of those user interviews, understand contextual meaning across text. Specifically, we
we propose to use LLMs to develop an interface to analyze the plan to develop an OpenAI powered chat-bot for its simplicity,
results from user interviews and generate summaries in a
versatility, and performance. This chat-bot takes user
simple and concise manner.
questions, queries documents for relevant information, and
In this research, we plan to develop a user-friendly chat generate responses that not only pull from the context it
system that allows service providers and product developers to learned during training but also from the client interview
better understand water usage clients’ needs and demands transcripts. OpenAI’s models are notorious for their ability to
related to saltwater intrusion. This proposed system uses understand complex pieces of text and extract details from
transcript obtained from interviewing a number of clients who them. This will see the chat-bot be able to offer insights
represent various organizations. The interview transcript data related to the wants and needs expressed by the clients in their
was anonymized to ensure no sensitive data is shared with interviews.
LLM providers. The interview data is fed to an LLM to
construct the proposed AI-powered chat system. The design of C. What is Langchain?
this product is inherently non-domain specific, allowing it to
theoretically be used for user requirement analysis in other Due to the rapid growth and popularity of LLMs and chat-
application domains. bots, user-friendly mechanisms for handling and working with
We have seen a growth in models designed to help these models have become necessary. LangChain has emerged
administrative and specialized industries constructing as one of the most popular frameworks, providing many
specialized LLM architectures. One example comes from helpful tools and components that make it quick and simple to
financial software company Bloomberg who is working on a incorporate complex, recent innovations, and strategies into
local LLM for their own financial usage. They have shown modern LLM-based applications.4 Our research aligns with the

987-1-6654-7345-3/22/$31.00 ©2024 IEEE

type of application intended for LangChain, as it requires these relevant documents. This approach disregards the context
tactics for efficient and effective LLM explanations. provided by previous interactions, underscoring the need to
incorporate prior conversations to maintain continuity from
III. BACKEND CHAT-BOT the end user's perspective.
A. Document Processing and Storage To alleviate this, we integrate what is known as memory.
In order to use and access documents given to the system, Memory in the realm of LLM powered chat-bots involves
it is necessary to define a consistent file format that we could storing a collection of previous prompts. When a new prompt
easily access and convert various possible interview document is generated, it is added to this collection, and the entire set of
formats into. A good choice is the simple plain text or .txt prompts is sent to the LLMs. This approach ensures that all
format. Therefore, we created an array of file converters to prior conversations are accessible to the LLM, thereby
turn possible file formats, such as .docx, .doc, .csv, .md, .xlsx, maintaining continuity and providing more contextually
.xls, and .pdf, into plain text. Documents were then split into accurate responses. We implemented these structures using the
sections, embedded by OpenAI’s text-embedding-ada-002 LangChain framework.
embedding model and stored within a local FAISS vector C. Prompt Engineering
database, as shown in Figure 1.
To keep the costs of running this program practical for
individual and small organizational use, smaller LLM models
are necessary. In our case, we opted for OpenAI’s GPT-4o-
mini model due to its significantly reduced price compared to
GPT-4 and GPT-4o. However, as the size of the LLM
decreases, so does its capability to store and understand large
amounts of context.
One beneficial feature of memory is the storing of a system
messages. Just as user and chat-bot conversation messages can
be saved, it is often advantageous to store the first message as
an explanation of the LLM's purpose and specific
requirements. This approach helps mitigate some limitations
of smaller models by clearly defining certain expectations. For
example, if identifying clients in a meeting is a central task for
the LLM, it can be specified that individuals are rarely
representatives of only themselves, thus the LLM should avoid
listing individual names. While system messages can be
utilized without storing them in memory, this requires they be
passed in on every query of the LLM, further taking up
valuable context space.
Figure 1. Structure of document loading system
Another advantage of introducing a system message is the
creation of personas. A persona is a role assigned to the LLM
By default, vector databases return the top k most relevant to help it better understand its purpose and how it should
documents, which lead to an unnecessarily large amount of interact with the user. These personas are included in the
data having to be processed by the LLMs. This is because, if memory via the system message before any prompts are sent.
there exists less than k documents that are relevant to prompt, In our case, it was beneficial to inform the chat-bot that its
the querying algorithm will begin to select irrelevant primary purpose is to explain client criteria and needs. This
documents. Documents upon retrieval must be scored based ensures that its responses focus on the clients' needs and
on relevance to allow for filtering the results to only include concerns, even if these are not explicitly stated by the user. 7
the most relevant ones.
In this project, the prompt constructed a persona of a
B. Chat-bot Construction document analyzing assistant. On top of personas it was found
To use our chat-bot backend, we process user input to be advantageous to set guidelines of what was needed from
prompts by querying the vector database to retrieve a set of responses such as giving follow up questions, and the length
relevant document sections. These sections are incorporated of inference that the LLM should make in regards to client
into the user given prompt, which is then sent to OpenAI for needs. We can also specify unwanted behavior discovered
processing. OpenAI generates a response based on the during testing of bot examples, including fabricating
enriched prompt, which we subsequently use as chat-bot information, mentioning source documents, and overly lengthy
outputs. explanations.
However, functional issues arise from this schema, the D. Specialized Memory
most significant one being the lack of contextual continuity in
conversations. For example, if a user encounters an issue and Token amount overall is a major problem in a setting like
requests further clarification on a specific point, the LLMs this. As handling of large amounts of information requires
only receives the current prompt along with a set of potentially increased computation and necessity for increased amount of
context to understand, therefore the high token count results in
increasingly less practical and more costly bots. Various
memory solutions have been applied to deal with this issue.
Two main ones are summary and entity memory. Firstly,
summary memory will store a summarized version of the
conversation that GPT appends new information to after each
part of conversation. The second form of memory is entity
memory, which over the lifespan of the conversation keeps
track of relevant “entities” which encapsulates a person, place,
thing, or idea that may be continuously referenced throughout
the conversation. Entities can then be queried similar to vector
database and appended to as new information comes out.
These forms of memory then allow for novel
implementation. An example in our research was investigating
using summarization memory to shrink documents into only
relevant details. This can be increasingly practical due to
documents high token count and large quantities of overly Figure 2. Example of chat interface
specific data such an individual’s name, specific times, etc.
E. Web Searching
The overall goal of the User Interface was to keep layout
Another problem the chat-bot may run into is a lack of clean, user-friendly, and fairly simplistic. Fortunately, we have
context in regards to some aspects of an interview. If a user seen large amounts of LLM-oriented applications and websites
wanted to know more about a specific organization which may have achieved similar goals effectively. The largest of these is
not be contained within the documents, the model would be OpenAI’s ChatGPT. Inspired by its user interface, we have
incapable to answer the question properly. Furthermore, the opted for a layout with a large chat box taking up majority of
user’s question may require up-to-date information that wasn’t the screen. This prevents visual noise from taking away user’s
available at the time of training for the LLM. To increase the attention and centers the application towards the chat.
LLMs ability to understand context, it is beneficial to give the
LLM access to documents outside of the ones provided. B. Conversation Loading

Our application has two main ways of handling this. Another practical feature we implemented was the ability
Firstly, when web searching is needed, our AI can request to save and continue previous conversations. The functionality
web documents via the DuckDuckGo API. The second is for this however is more complex and implemented in large
ArXiv’s search. ArXiv is an online academic and scholarly part by us. The way in which this works is to store the
documents database ran by Cornell University 6. It is highly conversations in a SQL database where every saved
beneficial to our chat-bot especially in areas such as Water conversation has a unique key that is generated during run
Salinity where the knowledge is often highly technical and time. These conversations then are loaded into the frontend
complicated and interview data is unlikely to go into those and reconstructed into the various chat messages and saved
technical details, and DuckDuckGo may not return academic into chat history for backend purposes. The frontend then is
resources. just an array of buttons allowing for user to choose desired
chat session, as shown in Figure 3.
IV. FRONT-END METHODS
A. User Interface
The front-end for our application has a few requirements
and goals to make it as simple as possible for the end user and
for future development. Firstly, we wanted the front-end to be
able to be used as many various types of applications, such as
a GUI Desktop Application and Website given its need to be
used within various areas of research and confine to their
needs. We also wanted fast development and therefore needed
a language that is easy to work with, this lead us to use the
Next.js web framework due to its robust, already-existing
infrastructure.
Figure 3. Chat selection user interface
 V. CLIENT INTERVIEW DOCUMENTS ANALYSIS are not clients themselves, making the results received more
consistent and accurate.
A. Client Identification
B. Need Identification
With the design of specialized chat-bots such as this, it
opens the possibilities for generation of specialized algorithms Once clients have been identified, the naive approach of
and structures to maximize the intended capabilities. One of identifying needs becomes significantly more logical. This is
the major ideas of this project was the development of because, given client names, very few documents are likely to
specialized documents that are easily understandable and pertain to each the smaller clients within the Salinity field. We
precomputed such that many of the questions likely to be did however discover major differences between the usage of
asked by users can be answered quickly. Given the client- smaller singular document based databases and large
oriented perspective, these generated documents will be a cumulative databases. We have chosen to be oriented towards
simple list of every client needs and the relevant clients. The cumulative databases to maximize information provided to the
LLM.
two parts identify the clients and list of needs of the client, as
shown in Figure 4. With this system, a document generation becomes simple
opting for markdown as preferred format allowing for both
Human readable and LLM readable format to be built with
very little required effort. These documents have since been
loaded into LLM.

Figure 4. Identified clients and their needs

The first section of this program was to extract data on who

Figure 5 Client needs example
each of the clients are. A naive approach for this is to simply
send a prompt to OpenAI requesting names of the clients from As seen in Figure 5 we see the output generated from this
documents. However, this solution has two major flaws. The structure overall the number of clients seems to line up with
first one is a limited number of documents that can be sent at expected output from meetings although with a repetition in
one time. Therefore, large amounts of data will not be seen USGS. Wants derived from the interviews also seem to line up
and cannot be analyzed. Second, due to the anonymization of with products and information seen in the transcripts. Overall,
data, people in meetings such as names like ‘interviewee1’ we have found these results to be satisfactory.
will be assumed to be the same person when it is very
probable to be different people in different interviews. VI. RESULTS
A. Experimental Results
One solution to this problem is to keep track of documents
stored and run through each document individually. This Overall, we have found that modern LLMs have become
requires independent vector databases for each document and very proficient at identifying and understanding complex
logging of which interviews exist. Another novel strategy to topics such as salinity. They can easily understand documents
identify clients is the usage of entity memory. The idea is that given to it and use information from those documents in
all clients are entities. Identifying clients will simply be response to user queries, as shown in Figure 6.
pulling entities from entity memory that are associated with a
given document. These structures though cause a secondary
issue, that being defining who the ‘client’ is in a given
document. In order to identify who the client organization is,
we must scan the documents and identify references to other
potential clients who are not actually present in the meeting.
Once identified the LLM can be informed that these references
 Figure 7 shows client needs generated on the second system
where only 1 of 5 clients were identified in the meeting.
C. Document Generation Findings

In regards to documents, the overall structure of extracting

clients followed by extracting each client needs works as a
good solution. However, specifics of the program such as
attempting to identify interviewees vs directly asking for client
show very different results. Our observations have shown that
the first system significantly reduces the number of clients
given. Such a system works extremely well on interview
datasets. However, using test dataset of Long Beach County
Figure 6. Example output of the developed chat-bot
City Council meetings where there exists larger amounts of
From the example output in Figure 6, we can see that the clients, the second strategy became more effective, as shown
Chat-bot is capable of taking a topic very specific to water in Figure 7.
usage and output relevant information about each of the clients
and specifics about them. The modern LLM architecture as
indeed hit a point where it has become powerful, malleable,
VII. CONCLUSION
and simple enough and specialized, locally-made systems can
be practical for organizational usage. We have found that LLMs have proven to be effective at
dissecting and understanding user interview documents. But
B. Chat-bot Findings we have observed that many of the recently created support
structures around them such as vector databases, entity
Throughout the research many discoveries were made in
regards to vector databases, web searching, and specialized memory, and at times summary memory have proved to be
memory. Firstly, we have found that there are significant less effective than desired to be used in a task such as this.
problems in regards to document querying. Given the Despite these shortfalls, overall, specialized chat-bots,
following prompt “Please give me an explanation of the needs particularly around client needs identification, could be a
of Maryland Department of Planning”, we found that on practical and fairly reliable tool that can be used by
default similarity search without score has a precision of 50% organizations on various products, research and future
of documents given were relevant. Followed by a recall of endeavors.
10% of relevant sections of documents were properly given.
The latter result is to be somewhat expected given the limited
amount of space. We have found that document scoring to be
ineffective at removing irrelevant documents as all retrieved
documents had good scores. REFERENCES
[1] S. W. Chang, T. P. Clement, M. J. Simpson, and K.-K. Lee, “Does sea-
Summary memory was found highly effective for level rise have an impact on saltwater intrusion?,” Advances in Water
conversations. Long conversations had little to no Resources, vol. 34, no. 10, pp. 1283–1291, Oct. 2011, doi:
https://doi.org/10.1016/j.advwatres.2011.06.006.
penalization. However, when used to summarize documents
before sending to AI, we saw significant decreases in [2] Water Resources, “Saltwater Intrusion,” www.usgs.gov, Mar. 02, 2019.
information kept in summarized documents. https://www.usgs.gov/mission-areas/water-resources/science/saltwater-
intrusion
[3] IBM. “What Are Large Language Models? | IBM.” Www.ibm.com,
2023, www.ibm.com/topics/large-language-models.
[4] “LangChain.” Www.langchain.com, www.langchain.com/.

[5] S. Wu et al., “BloombergGPT: A Large Language Model for Finance,”

Dec. 2023. Available: https://arxiv.org/pdf/2303.175
[6] “arXiv.org e-Print archive,” Arxiv.org, 2019. https://arxiv.org/
[7] G. Sun, X. Zhan, and J. Such, “Building Better AI Agents: A
Provocation on the Utilisation of Persona in LLM-based Conversational
Agents,” Jul. 2024, doi: https://doi.org/10.1145/3640794.3665887
Figure 7. Example of Long Beach client needs