Monitoring
Once your version of Loome Assist is setup, and begins talking with users, reporting will be available to provide insight on how it is being utilised. This can take the form of scorecards with key metrics such as #Sessions and %Correct Answers, or even be as simple as tracking every interaction with Loome Assist in a conversation log.
These reports are built in Power BI, a dynamic tool that can tailor the reporting you need to optimise Loome Assist going forward. Here are a few examples of what Loome Assist captures and what you can do with it;
Scorecards: Rolled up metrics that look at overall activity, such as how many interactions per day over a given time frame, the number of unique users and questions they ask, and the average responses per question. These are usually for high-level reporting purposes, created with the ability to drill-down for more analytics below.
Response Feedback: Visualising how Loome Assist is responding to queries, such as; How many times did users request contact info? How many asked for help? Did questions trigger a normal answer or an action flow? Was there a ‘no’ response? Were escalations made? Feedback monitoring can be used as another method of not only tracking usage, but in filling any gaps in Loome Assist’s knowledge base and, importantly, how he responds to any updates you have made.
Knowledge Base Refinement: As an extension of the above, reporting can also observe the most common keywords and questions entered by users. What tags are being matched to the most common questions? What tags a matching to more specialised questions? Which responses is Loome Assist giving by tag set (noun/verb/query)? Are there multiple responses being generated due to scoring? This can be used to further refine your knowledge base to optimise your tagging, responses and answer grammar/phrasing.
Keyword Tag Analyses: The reporting can also provide a clearer summary of the words you have in your Dictionary and the aligned tags in your Knowledge Base. As these libraries grow, it can be hard to manage (and prevent possible duplication) using the normal UI, so Power BI steps in to display this for you. For example, all nouns, verbs, queries, and their types can be observed and drilled into, to provide insight in managing these areas.
Error and Conversation Logs: As mentioned above, Power BI can also produce conversation level reporting. Not only used for granular analysis/investigation, this can be very handy in tracking errors - such as Loome Assist producing a no response or an incorrect failover to another service, so that you can promptly correct.
Note: Reporting using the metrics above will be available very soon. Once released, you can then customize the data according to your needs.