Research and web application
Evaluation and redesign DMS
for Municipality of Eindhoven
for Municipality of Eindhoven
Goal
Explore reasons for low acceptance of the document management system. Propose design solutions to increase its usage.
Result
High-fidelity prototype and list of process and product requirements with priorities
Below I will describe the process of work that my team applied in service of the Gemeente Eindhoven in March-April 2016 concerning the evaluation and redesign of the document management system SilverDocs.
This process explains the used methodology, findings and further advise.
This process explains the used methodology, findings and further advise.
Research
Initial analysis
At the start of this project we were able to participate in a training of SilverDocs at Gemeente Eindhoven. During this training we observed the participants, coded their statements and observed the methods of working with SilverDocs taught during the course.
Errors, misunderstandings and usability issues of participants and trainer were noted and transcribed into:
Errors, misunderstandings and usability issues of participants and trainer were noted and transcribed into:
36%
Usability / Redesign
30%
Structural
20%
Security
14%
Behavorial
The top row represents desired, positive outcomes
of using the system. The middle rows represent the features present in the system and the bottom row represents occurring negative outcomes in the current situation.
Features underlined in red are currently blocking positive outcomes from being realized.
of using the system. The middle rows represent the features present in the system and the bottom row represents occurring negative outcomes in the current situation.
Features underlined in red are currently blocking positive outcomes from being realized.
User testing
We recruited our participants from the employees of different departments in Eindhoven Municipality. There were 7 participants.
Three of the participants were everyday users of SilverDocs and had a significant experience with the system; others did not have such experience and used the system occasionally. This combination of users has provided us with insight into both novice and expert behavior.
Below is the scenario of the user test:
Three of the participants were everyday users of SilverDocs and had a significant experience with the system; others did not have such experience and used the system occasionally. This combination of users has provided us with insight into both novice and expert behavior.
Below is the scenario of the user test:
Questionnaire
After performing the tasks participants were asked to fill in a questionnaire. The questionnaire contains 36 questions, each requires a marking on a 5-point Likert scale as an answer. The questionnaire is adapted to use from Factors Influencing Employees' Intention to Use an Electronic Recordkeeping System: Development of a Valid Survey Instrument, which combined multiple questionnaires such as Technology Acceptance Model, Unified Theory of Acceptance and Use of Technology Model and others.
Internal Factors
Internal Factors
- Technology Acceptance
1. Performance Expectancy
2. Effort Expectancy
- Organizational Context
3. Perceived Power Security
4. Social Influence - Knowledge Interpretation:
5. Perceived Value of Records
Dream Scenario
The next step in the experiment was to ask participants to imagine the perfect document management system that might replace SilverDocs. This information helped us to discover some higher-level usability problems and answer research question Q2 while also gathering valuable input concerning attitudes towards SilverDocs and the integration of SilverDocs within the participant's work.
List of tasks
Based on the core functionality:
- Document import (From Disk and from Mail);
- Document search;
- Dossier management;
- Version control & Document history;
- Workspaces, personalization and saved searches;
- Assignment of the document/dossier to a person or group.
Ranking
The experiment concluded with the participants rating the functionalities of the system based on which features they experience as being the easiest/hardest and least used/most used. This information helped us gain insight into research question Q3
Results of Analysis
Questionnaire
Results
Results
Our three main findings from these results:
- First, all the users have a clear understanding that managing documents is an important task that requires everyone's attention.
- Second, the users whose job is to manage large numbers of files and documents rarely use SilverDocs.
- Finally, we found that the users do not feel influenced by their colleagues and their organization to use SilverDocs. Accepting the organizational policies regarding usage of SilverDocs might facilitate the user acceptance of the system.
Dream Scenario Results
When speaking about dream document management systems, many participants mentioned the advanced filtering and personalization of the system. Another trend is a wish for leading, guiding system.
Finally, the participants mentioned the search problems several times.
The results of the dream scenario evaluation show that there is a clear need for a different model of working with the documents, more guiding, personalisable and equipped with a powerful and reliable search engine.
Finally, the participants mentioned the search problems several times.
The results of the dream scenario evaluation show that there is a clear need for a different model of working with the documents, more guiding, personalisable and equipped with a powerful and reliable search engine.
“
I would like more clear manuals
for the functions most used by me
for the functions most used by me
“
Would only show what is useful for me
“
Lead me through what I need to do
Task Performance
& Ranking Results
& Ranking Results
We transcribed the video recordings and combined these with our notes in order to perform a content analysis on the transcript and analyse all the errors, confusions and misunderstandings that the participants encountered. As can be seen below, most errors were made while importing a document. The second task that obtained many errors was dossier management.
Issues grouped by functionality based on the results of coding errors of participants
The most frequent and most difficult ones appeared to be the ones, where the most errors appeared:
— Document Import and
— Search
— Document Import and
— Search
"Document import" and "Search" appeared to be categories, which caused the highest amount of problems and also they were the most frequently used ones. Moreover category "Workspace" is an essential part of the SilverDocs interface itself and category "Metadata & Assignment" is a part of "Document Import".
The scope of our further research was based on these results.
The scope of our further research was based on these results.
Low-fi prototype
Paper Prototype
We created paper prototypes for two scenarios: import a document and perform a search.
List of changes
- Repositioning of main elements -> more space for essential content, such as search results;
- Hiding less frequently used fields -> less visual noise, preventing confusion caused by number of unnecessary fields, more space for essential content;
- Feedback (highlighted required fields, button statuses) -> gives users clearance of system behavior by responding to their actions;
- Make shortcuts visible -> proposing users different possibilities of importing one or more documents, which would increase a pace of working of more advanced users;
- Increase visibility of system status by showing highlighted results, settings, logout, headings;
- Adding a column with dossier to a search results -> allows users to have a bigger picture;
- Adding a column with a type of the document -> allows users to sort by document type;
- Plain language feedback -> offers simple error handling by explaining the problem
Evaluation
To validate our findings we ran two sets of testing:
1. Heuristic Evaluation
- 5 participants
- Experts (User System Interaction PDEngs)
- Not familiar with SD
For the first evaluation we used Shneiderman's "Eight Golden Rules of Interface Design". Based on specific tasks, participants were asked to evaluate our prototype on a 5-point Likert scale and added an explanation or a comment for each heuristic.
2. Co-discovery
- 4 participants (2 pairs of colleagues)
- Gemeente employees
- Familiar with SD
For the second evaluation we used the Co-discovery Learn method. Participants had to talk aloud naturally to each other whilst completing tasks together.
Hi-fi prototype
