Effective and Usable Validation for Long Term Data Entry and Workflow

This topic sure sounds droll! But I've been thinking about it a lot as we've dealt with these issues at my company, so I thought I'd do a brain dump.

In large applications that store complex data, validation of form input is likely going to be more involved than just checking the fields' existence and format on submit. Many times the data comes in in bits and pieces, over a long period of time, and is entered by several different users, before the form is "complete". The validation rules may also have too many exceptions and special cases to really enforce them. This probably comes back to poor process and application design, but it certainly exists and you may just have to live with it. However, some type of validation needs to be in place to maintain data integrity - whether its for reporting, data mining, or simple assurance that the information in the database is accurate.

One approach for validating this "long term" data entry is to attack it in stages or tiers, with each progressive stage becoming more stringent, until the final step at which everything must be complete before (figuratively) closing the file. Here's an example.

Stage 1 - Alerting the user to data that is needed.
Non obtrusive alerts serve to bring the user's attention to parts of the form that they "may want to fill out" before moving to the next screen. This warning should be clear in its purpose, to essentially let them know, "at this stage in the process, you usually want these fields to be complete". This might be some type of border, icon or label combination that indicates important fields. The warnings will only apply to this particular step of the workflow. There may also be different levels of warnings - such as "information alert", then "warning", then "critical error" - with different rules as to whether or not the workflow can continue with their presence. If future steps in the workflow are dependent on the existence of certain data, that would usually mean that it's a "critical error".

Step 2 - Warning the user that data is needed.
Now the user is attempting to move forward in the workflow, and data is still missing. When the user hits the button to send the form on to the next stage in the Workflow, they will see a validation warning - a pop up window or screen - that says "these fields should be filled out before moving to the next stage". The missing fields should also be highlighted in such a way that the user can easily see what's missing, perhaps with an error message next to them. At this point, you may want to prevent them from moving forward until the data is complete, or allow them after they acknowledge the warning. This depends on the situation and the stage in the process. The warnings usually only apply to the current section in the workflow, and may also follow a scale of severity.

Step 3 - Preflight before closure
When the file is approaching closure, as the final step, there is a preflight stage, where a list of exceptions or warnings is presented to the user before closing the file. Up until now, the only warnings shown apply to the current step in workflow. Now the preflight "issues list" will show exceptions from every stage, organized by stage, and ideally give the user a single, simple place to fix these issues (such as a custom wizard/form that walks them through completing each field step by step) before proceeding. As with the other errors, the preflight will show exceptions on a severity scale, from "information" to "warning" to "critical error". You can decide how to handle each level of warning, but typically warnings "should" be fixed, and "critical errors" will prevent the file from closing.

Step 4 - Post close audit
Depending on your business rules, files may have closed with information still missing. Therefore, you will want to have an audit report (really, it should be available at any step in the process) that will show each exception allowing errors or missing data to be easily found and corrected. You may also wish to display this information with the individual users at each step in the workflow, so that you can tell who sent this file to the next stage with incomplete data.

A good working example of this type of validation is consumer tax preparation software such as TurboTax. Building flexibility into the data entry process, and allowing the user to move forward even with incomplete data can dramatically improve usability - required fields are the user's enemy - but there need to be checks and balances to ensure data integrity is maintained.