New Bight ’18 tools to improve data management workflows

SCCWRP has developed a series of new tools and automations for the 2018 cycle of the Southern California Bight Regional Monitoring Program designed to streamline and enhance data management.

Known as an open data workflow, the data management solution will eliminate the cumbersome version-control issues associated with sharing databases and performing analyses and quality-control checks.
Instead of requiring participants to manually download databases as ZIP files from a SCCWRP-maintained server, and then analyze a non-master copy before reuploading the database, the open data workflow will enable participants to work directly from a single copy maintained within the cloud-based data portal.

Furthermore, all edits and changes to a database will be tracked in the data portal, giving all participants visibility over data management activities.

The open data workflow represents a significant advance over the data management approaches used for previous Bight programs, with widespread applicability to other data management projects beyond Bight regional monitoring.

A new, dynamic quality-assurance/quality-control checker also will be built directly into the data portal, allowing all custom data and business-rule checks to be completed at the same time participating agencies upload their data.

The open data workflow gives Bight ’18 the option of integrating multiple additional features, including descriptive statistics, custom calculations and indices. Additionally, graphic and map-based visualizations can be developed and published directly from this dynamic system, without the necessity of downloading and reformatting the data first.

Finally, databases and other outputs can be directly streamed to SCCWRP’s new open data portal and other federated data sites, ensuring the data become more easily discoverable and accessible.