Designing a Scalable Data Management System

To avoid duplication of data processing and improve knowledge sharing across evaluations, rowsquared conceptualised a SharePoint-based Data Management System (DMS) for the German Institute for Development Evaluation (DEval). We structured the DMS around a clear separation between standardised, core datasets and project-specific workspaces, letting teams work independently while benefiting from shared resources and metadata standards.

A comprehensive needs assessment with DEval revealed two areas of improvement: firstly, each team at the DEval cleaned and processed data in isolation, working with the same data but repeating processing steps and secondly, a lack of knowledge sharing across evaluations.

We proposed a SharePoint-based DMS built around a centralised “Core Data Hub” containing frequently used, pre-processed datasets, and separate “Project Hubs” for evaluation teams. We developed a metadata framework and access protocols, ensuring data traceability, versioning, and role-based permissions. We also proposed low-code automation using Azure Data Factory (ADF) and Power Automate to ingest, transform, and validate commonly used datasets, reducing manual processing time.

To support data reuse, we proposed to integrate a data catalogue solution using Microsoft Purview, allowing teams to search across projects, link external datasets, and manage metadata centrally.

Our DMS concept is now guiding DEval’s approach to data management and lays the groundwork for standardised workflows and AI-supported analysis.